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Preface 


In 2013, NYC Leadership Academy (NYCLA) received funding from the U.S. Department 
of Education through the Investing in Innovation program to develop the Targeted Intensive 
School Support (TISS) program and contracted with RAND to conduct an independent eval- 
uation of the program. This report describes the findings on the implementation and effects 
of TISS from the evaluation. The research was undertaken by RAND Education and Labor, a 
division of the RAND Corporation that conducts research on early childhood through post- 
secondary education programs, workforce development, and programs and policies affecting 
workers, entrepreneurship, and financial literacy and decisionmaking. The research reported 
here was supported, in whole or in part, by the U.S. Department of Education (under grant 
number U411C130107) through a subgrant to the RAND Corporation from NYCLA. The 
opinions expressed are those of the authors and do not represent the views of NYCLA or the 
U.S. Department of Education. 

More information about RAND can be found at www.rand.org. Questions about this 
report should be directed to Idaugher@rand.org, and questions about RAND Education and 
Labor should be directed to educationandlabor@rand.org. 
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Summary 


Effective leaders can play an important role in the success of a school. As a result, districts and 
other educational organizations have increasingly invested in programs preparing new lead- 
ers and supporting these leaders as they enter schools. In 2013, the NYC Leadership Acad- 
emy (NYCLA) developed a leadership intervention in collaboration with the New York City 
Department of Education (NYC DOE) to support schools that were facing particular chal- 
lenges, the Targeted Intensive School Support (TISS) program. NYCLA asked RAND to pro- 
vide an independent evaluation of the program’s implementation and effect; this report details 
findings from this evaluation. 


The Targeted Intensive School Support Program 


Prior to TISS program, NYC DOE had a robust leadership support system. The district had 
internal preservice preparation programs for principals and assistant principals (APs) and part- 
nerships with nonprofit organizations like NYCLA and universities to provide other oppor- 
tunities for preservice training. New principals entering NYC DOE were provided with the 
Opportunity to access up to 72 hours of coaching in the first year at no cost to the school. 
This coaching was initially provided by NYCLA, but was brought in-house to create a district- 
provided coaching program over the course of the study period. Principals in NYC DOE 
schools were also supported by strong evaluation processes and a variety of professional devel- 
opment resources. 

The TISS program aimed to adapt the traditional individually oriented model of leader- 
ship development and support to establish a team-based approach to leadership for NYC DOE 
schools that were facing particular challenges. The program was designed to have the following 
five key components: 


1. Teaming and shared preservice training: The program aimed to match principals and APs 
from NYCLA’s existing preservice programs based on leadership styles and engage par- 
ticipants in a range of collaborative preservice training activities to build their capacity 
as a team. 

2. Coplacement into NYC DOE schools: The program aimed to coplace these principals 
and APs with shared training into a school, providing the principal with a like-minded 
leader who could help to execute improvements to the school. 

3. Team-based coaching: The program aimed to provide coaching to both the principals 
and the APs (and in some cases the broader leadership team), with an expectation that 
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30 percent of all coaching hours be used to support others on the leadership team in 
addition to the principal. 

4. Extended coaching: The program aimed to increase the number of hours of coaching and 
the length of time over which coaching took place. Although new principals in NYC 
DOE traditionally were offered 72 hours of coaching in the first year of principalship, 
the program aimed to provide school leaders with 328 hours of coaching over the first 
three years. 

5. Needs-based coaching: The program aimed to engage the principal and other school 
leaders in a diagnostic process to review data and identify the schools’ priorities and 
areas for needed support. The expectation was that coaching would then be informed 
by these identified priorities and support needs. 


TISS leaders were drawn from NYCLA’s existing preservice training programs for principals 
and APs. The program served a total of 32 NYC DOE schools over five years: the first set of 
TISS leaders were placed into schools prior to the 2014-2015 academic year, followed by two 
additional cohorts entering in 2015-2016 and 2016-2017. After entering NYC DOE schools, 
TISS leaders were supported by NYCLA for up to three years. 

The implementation of the program’s key components evolved over time due to changing 
conditions on the ground. Because of the timing of when the program was initially funded, the 
preservice component of TISS could not be fully implemented for the initial cohort of TISS 
leaders, with later TISS cohorts benefiting from more information on the program, expanded 
matching and teaming activities, and earlier exposure to the coaching and diagnostic com- 
ponents. In the second year of the program, it became clear that the district was not going 
to provide any specialized placement preferences for TISS principals, so NYCLA loosened its 
restrictions on coplacement to sustain the program and continue serving as many TISS par- 
ticipants and schools as possible. To support these principals who were not coplaced, NYCLA 
had to adapt its conception of team-based coaching to focus on the identification of partners 
within the existing school team rather than centering on an AP partner who the principal was 
paired with prior to placement. There were also several other adaptations to the program over 
time, including the relaxing of requirements that TISS participants complete the preservice 
program and the embedded matching and teaming components, and greater formalization of 
and emphasis around the diagnostic component. 


Study Approach 
To assess the implementation of the TISS program, RAND worked with NYCLA to establish 


fidelity measures and thresholds for each of the program’s key components. Drawing on data from 
coaching records and other program documentation, we tracked fidelity of implementation across 
the full study period for each school where a TISS principal was placed. When 75 percent of TISS 
schools met fidelity threshold(s) for a key component, TISS was determined to have been imple- 
mented with high fidelity at the program level. The implementation analysis also documented the 
adaptations of the program and the barriers and facilitators to implementation through interviews 
with TISS principals, TISS coaches, NYCLA staff, and district leadership. 

To assess the effect of TISS, we first needed to identify a set of comparable principals 
and schools. Given that all TISS principal participants were new principals and were expected 
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to come from NYCLA’s residency-based preservice program—the Aspiring Principal’s Pro- 
gram (APP)—we needed to compare TISS principals to other new principals participating in 
residency-based training programs. Some of these comparison principals were trained through 
APP but opted not to participate in TISS or were trained in earlier cohorts, whereas other 
comparison principals were drawn from other residency-based principal preparation programs 
offered by the district or external providers. We required that principals remain in the school 
for one full year to be included in the sample and continued to include the school in the sample 
even if the TISS principal left the school. 

Once narrowing the sample to schools with new, residency-trained principals who 
remained in the school at least one full year, we used a propensity weighting approach to ensure 
that the comparison schools looked similar to TISS schools in terms of academic achievement 
prior to the new principal’s placement in the school. To be included in this weighting process, 
schools had to have one year of preplacement data; a total of 28 TISS schools and 54 com- 
parison schools met these requirements. After weighting, the comparison group schools were 
similar to TISS schools according to prior achievement in English language arts (ELA) and 
mathematics, as well as the demographic and socioeconomic characteristics of their students. 

To account for the small remaining differences after weighting, we used a generalized 
linear regression model that controlled for a number of school-level characteristics, including 
prior test scores and chronic absenteeism rates, school grade configuration (e.g., high school, 
middle school, and K-8 school), and the proportions of students by race/ethnicity, disability 
status, English Language Learner status, and socioeconomic status. We estimated the effects 
of the TISS program on average school outcomes in the third year after the TISS principal was 
placed. The primary outcomes were ELA and math achievement scores, though we also con- 
ducted exploratory analysis examining effects of the TISS on chronic student absenteeism, six 
dimensions of school culture, and principal retention. 

There were several limitations to the study approach. First, with just 28 TISS schools and 
54 comparison schools, we had limited power to assess effects and examine variation in effects 
by implementation and across key subgroups of interest. Second, TISS principals were not 
selected randomly and the comparison group included schools and principals who may have had 
different characteristics than TISS principals and schools. Therefore, the differences observed 
in the outcomes of TISS and comparison schools may not be fully attributed to the TISS pro- 
gram. In addition, evolving treatment and comparison conditions over the study period made 
it difficult to characterize the program and the counterfactual, creating some challenges with 
interpretation of the findings. And finally, we faced data limitations that prevented us from 
examining some aspects of implementation and effect that we had hoped to assess. 


Implementation Findings 


Only two of the five key components of TISS were implemented with fidelity (Table S.1). 
The majority of TISS principals completed the APP program, though the exact teaming and 
matching activities participants engaged in during this preservice component varied widely 
across years and across participants within a given year. The diagnostic component was also 
implemented with fidelity, with 82 percent of TISS schools completing a diagnostic and using 
findings from the process to inform goals for school improvement and coaching priorities. 
Coplacement rates decreased dramatically over time, with only 50 percent of all TISS schools 
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Table S.1 
Percentage of Schools Implementing TISS Program Component with Fidelity 


Year 2 TISS Schools | Year 3 TISS Schools 
(2015-2016) (2016-2017) Total 


Year 1 TISS Schools 
(2014-2015) 


Coplacement 50% 11% 50% 


Extended coaching 71% 42% 
Needs-focused coaching 71% 
Total schools 7 


NOTE: Numbers represent percentages of schools in the final analytic sample implementing with fidelity. Cells 
highlighted in green indicate that the component was implemented with fidelity at the program level, meaning 
that at least 75 percent of schools are implementing the component with fidelity. Cells in red indicate that the 
component was not implemented with fidelity at the program level. 


receiving a TISS principal and AP who were paired and cotrained. The team-based coaching 
and extended coaching components were not implemented with fidelity, and there was wide 
variation in the amount of training received and the degree to which this coaching was team 
based across TISS schools. 

Based on interviews we conducted with principals, coaches, and NYCLA leadership, the 
program benefited from buy-in among coaches and some participants, strong support from 
program staff, and the flexibility of NYCLA in adapting the program to face challenges on 
the ground. However, these interviewees also reported substantial barriers to implementation 
that ultimately hindered NYCLA’s ability to implement the program with fidelity, including 
a lack of buy-in from district leadership and some TISS participants, limited understanding 
of the program among some stakeholders, major challenges coplacing principals and APs, and 
limited time among some TISS participants to engage in the additional hours of coaching. 


Impact Findings 


We did not find evidence of positive effects of the TISS program across the report’s outcome 
measures. In mathematics and ELA, we found no statistically significant differences in stan- 
dardized achievement scores when comparing TISS schools to the weighted group of com- 
parison schools with residency-trained principals (Figure S.1). We also found no statistically 
significant differences relative to comparison schools with residency-trained principals in the 
areas of school culture and principal retention. TISS schools underperformed relative to the 
comparison schools in terms of chronic absenteeism among students. 


Discussion 


The TISS program’s team-based leadership approach to supporting high-needs schools offered 


an innovative approach to leadership development and support in a context where most of 
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Figure S.1 
Third-Year Effects of TISS on Student Achievement Relative to Comparison Schools 


ELA Mathematics 


-0.02 


-0.04 


-0.06 


Effect size 


-0.08 


-0.1 


-0.12 


-0.14 


SOURCE: NYC DOE Administrative Data (AY 2015-2016, 2016-2017, 2017-2018, 2018-2019). 
NOTE: No results were statistically significant at the 5 percent level. Estimates represent standardized effect 
sizes. Estimates are based on 28 treatment schools and 54 comparison schools. 


these efforts were focused on the principal as an individual. However, NYCLA faced substan- 
tial challenges coplacing principals and APs, due in part to diminishing buy-in and support 
from the NYC DOE for the program as the district made changes to move away from 
external partners to provide more of their leadership support in-house. Despite NYCLA’s 
efforts to reenvision teaming in the face of challenges with coplacement, the TISS program 
was ultimately not implemented as planned. In light of this, it was not surprising that we 
found no evidence of positive effects of the program, with TISS schools underperforming com- 
pared with comparison schools with residency-trained principals in reducing students’ chronic 
absenteeism. 

These findings offer several lessons to districts and leadership support organizations. 
First, interventions that hinge on coplacement should be considered with caution because 
very specific conditions are needed for coplacement to succeed such as strong buy-in from the 
district, which may change with turnover in leadership and evolving priorities. When coplace- 
ment does not occur, the investments made in matching and cotraining teams of leaders prior 
to placement are lost. The evolution of TISS to focus on team-building among existing staff 
may have helped to build transferrable skills around teaming that were robust to where and 
with whom principals were placed. Flexibility and willingness to adapt interventions can be 
valuable in sustaining programs in district contexts with uncertain and changing conditions. 
However, in the case of TISS, such adaptations were not sufficient to overcome the barriers to 
implementation and generate positive effects for students. Finally, although TISS is unlikely 
to continue as a program, there may be some valuable aspects of the intervention that could be 
integrated into other leadership development and support interventions, such as the diagnos- 
tic process, which was reported to have helped TISS participants and coaches engage in more 
strategic coaching and leadership activities that align with priorities for school improvement. 
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CHAPTER ONE 


Introduction 


Effective leadership is an important component of school success. Principals are tasked with wide- 
ranging responsibilities. They establish a vision for the school, develop and maintain a school cul- 
ture, provide instructional leadership, evaluate and support the development of school staff, and 
ensure the school operates effectively and that facilities are maintained. Prior research indicates 
that there is variation in effectiveness across principals, and that principals’ skill in managing these 
responsibilities is related to student achievement (Bartanen, Grissom, and Rogers, 2019; Branch, 
Hanushek, and Rivkin, 2012; Hallinger, Bickman, and Davis, 1996; Supovitz, Sirinides, and May, 
2009; Walsh and Dotter, 2018), teacher retention (Boyd et al., 2011; Grissom and Bartanen, 2019), 
and other outcomes of interest (Liebowitz and Porter, 2019). 

Districts, states, and other support organizations are interested in developing systems 
and interventions that support principal effectiveness. Research points to a number of ways 
that such systems and interventions might influence principal effectiveness, including rigor- 
ous preservice preparation and support for early career principals once on the job (Gates et al., 
2019a, 2019b). States and districts use a variety of training models to help novice principals 
learn and manage the complex demands of the principal position, such as mentoring, coaching, 
new principal institutes and networking support (Crow and Whiteman, 2016). However, there 
remains limited rigorous evidence on which types of support for early career principals might 
be most valuable in supporting improved principal effectiveness and student achievement. 

This report documents findings from a rigorous evaluation of an innovative early career 
leadership intervention that was implemented in New York City Department of Education 
(NYC DOE) schools between 2014 and 2019. The Targeted Intensive School Support (TISS) 
program—a team-based leadership and coaching intervention—was developed to support 
turnaround in struggling schools by the NYC Leadership Academy (NYCLA). At the time 
TISS was developed, NYCLA was already operating preservice preparation and coaching pro- 
grams to support school leaders in NYC DOE schools. The TISS program aimed to draw 
participants and coaches from these programs to participate in its team-based approach to 
school leadership support, including cotraining and coplacing principals and assistant prin- 
cipals (APs) and 328 hours of high-quality coaching for the leadership team over three years. 

We start by providing some additional background on the context for leadership support 
in NYC DOE in Chapter Two. In Chapter Three, we describe the design of the TISS program 
and the prior evidence that suggests that the program’s components might help to improve stu- 
dent outcomes. We then provide an overview of our approach to analyzing the implementation 
and effect of the program in Chapter Four. Implementation and impact findings are summa- 
rized in Chapters Five and Six. Finally, we conclude with a discussion of the findings and key 
takeaways for leadership support efforts. 


CHAPTER TWO 


The Context for Leadership Support in New York City 


The NYC DOE is by any measure the largest school district in the United States. As of 2019, 
it served 1.13 million students, operated 1,840 schools and had an annual operating budget of 
$34 billion (NYC DOE, webpage, undated-a, undated-c). In 2017, NYC DOE had the high- 
est spending per pupil among the 100 largest districts in the United States with an average of 
$25,199 per student (U.S. Census Bureau, 2019). NYC DOE has typically outperformed other 
urban districts in the state: 47 percent of NYC DOE students tested proficient in English lan- 
guage arts (ELA) and 43 percent tested proficient in mathematics on New York State assess- 
ments in 2018, compared with fewer than 30 percent in other urban districts in New York 
(NY State Board of Regents, 2018). Like most school districts, NYC DOE has continued to 
see racial/ethnic disparities in student achievement. In 2018, more than half of White students 
scored proficient on English Language tests compared with 35 percent of African American 
students and Hispanic students. In mathematics, 54 percent of White students scored profi- 
cient, while fewer than one-third of African American and Hispanic students scored proficient 
(NY State Board of Regents, 2018). 

As the senior leader in the NYC DOE, the chancellor was the ultimate person responsible 
for leadership development. The district had layers of oversight and support between the chan- 
cellor and the school principal. By state law, superintendents were responsible for appointing 
and evaluating school principals. As of 2019-2020 school year, NYC DOE had 46 superin- 
tendents, who oversaw geographically oriented districts or groups of specialized schools within 
the NYC DOE. Those superintendents were, in turn, overseen and supported by nine execu- 
tive superintendents who have overarching responsibility for their region or group of schools. 

In hiring and supporting school leaders, superintendents and executive superintendents 
leveraged DOE-wide infrastructure including a district-wide pool of principal candidates and 
a number of support programs and career development opportunities for aspiring and current 
principals. To supplement the district’s internal capacity, NYC DOE also partnered with uni- 
versities and nonprofit organizations to provide leadership support through preservice training 
and support for early career principals. 

NYCLA has been a particularly important external partner to the district. NYCLA was 
originally established to support leadership development in NYC DOE, but has now expanded 
their course offerings across several states (NYCLA, webpage, undated-a, undated-b). NYCLA 
clients include school districts, state agencies, universities, and nonprofit organizations. Since 
its establishment in 2003, the organization has provided preservice training to aspiring prin- 
cipals and APs, coaching to new and continuing principals, and consulting and professional 
development around leadership support. Three of NYCLA’s main leadership support programs 
in NYC DOE are described in Box 2.1. 
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Box 2.1: NYC Leadership Academy’s Leadership Support Efforts 


Below we provide a brief description of NYCLA’s three leadership support programs in 
NYC DOE. These programs provided the foundation for TISS. 


Aspiring Principals Program 

The organization’s flagship program was the Aspiring Principals Program (APP), a preser- 
vice program for aspiring principals that was initially developed for NYC DOE but has 
now been replicated in other districts across the country. The program consisted of rigor- 
ous recruitment and selection to identify promising leaders, a five-week summer intensive 
program, a school-based residency under a mentor principal, and a summer planning phase 
to help principals transition successfully into their new schools. 


Leadership Advancement Program 

In addition to preservice training for principals, NYCLA also offered preservice training 
for aspiring APs through the Leadership Advancement Program (LAP). The LAP program 
consisted of weekly trainings and residency-based learning opportunities for teachers who 


aspired to be APs. 


NYCLA Coaching 

In 2005, NYCLA began providing coaching to most of the first-year principals in NYC 
DOE. NYCLA’s coaching program matched new principals with coaches—retired princi- 
pal supervisors with specialized training—who offered one-on-one support throughout the 
first year. NYCLA’s model for coaching was organized around four pillars: (1) a facilitative 
learning process whereby school leaders use inquiry, reflection, and feedback to shift their 
leadership behaviors; (2) coaching work that is closely aligned to leadership competencies 
and standards; (3) responsiveness to district policies and initiatives; and (4) tailored sup- 
port to meet individual needs and unique school context. All new principals were offered 
72 hours of coaching support in their first year. Continuing principals who wanted to pur- 
chase additional hours did so with their school budgets, and according to NYCLA, nearly 
50 percent of early career principals purchased additional coaching at some later point 
in their tenure. As of 2013, NYCLA had served more than 2,000 principals. However, 
as described below, NYC DOE began to make shifts in its leadership support portfolio 
under Chancellor Farina, and as a result, the district shifted away from NYCLA as the pri- 
mary provider of coaching for new principals. As of 2017, all coaching (outside the TISS 
program) was conducted by a district-provided coaching program, which was informed 
by NYCLA’s coaching program. NYCLA continued to provide coaching-related services 
to districts across the country and to experienced principals in NYC DOE interested in 
coaching. 
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The idea for the TISS program was developed by NYCLA in 2013 in collaboration with 
NYC DOE under Chancellor Dennis Walcott’s administration. However, implementation of 
the program occurred during the administrations of Chancellors Carmen Farifia (2014-2018) 
and Richard Carranza (2018-2019). Throughout the study period, effective school leadership 
was a priority for the NYC DOE, and there were a number of notable shifts in the systems 
and structures for supporting school leaders driven by changes in district leadership and NYC 
DOE’s involvement in the Principal Pipeline Initiative’ between 2011 and 2016 (Gates et al., 
2019b; Turnbull et al., 2016). In 2014, Chancellor Farifia advanced a new vision plan focused 
on “ensuring that every neighborhood has high-quality schools and that every child has the 
opportunity to succeed” (NYC DOE, 2015). To support this vision, the NYC DOE devel- 
oped the Framework for Great Schools emphasizing the importance of school principals and 
their ability to promote collaborative leadership within the school and to effectively engage 
with the community (NYC DOE, 2015). NYC DOE also promoted greater autonomy for 
principals regarding budget and staffing and restructured school and school leader support 
systems to provide a single point of contact in the areas of instruction, operations, special 
education, and student services such as health and safety. And importantly for NYCLA’s 
programs, including TISS, during this time, NYC DOE moved away from the use of exter- 
nal organizations to provide leadership support and toward more internal district responsi- 
bility for these efforts. 

Below we provide some additional details on NYC DOE'’s approaches to leadership sup- 
port over the study period in three areas most relevant to the TISS program: preservice leader- 
ship training, hiring of leaders into schools, and coaching and support for new leaders. 


Preservice Training for Leaders 


To serve as a school principal or APs in NYC DOE public schools, aspiring leaders had to 
obtain an administrative license from the State of New York. This required completion of 
a state-approved preparation program, which included several district-provided programs, as 
well as universities and nonprofit organizations that the district had established partnerships 
with (Gates et al., 2019a). External preservice preparation providers explicitly aligned prepa- 
ration efforts with the expectations for NYC DOE leaders. Aspiring leaders were responsible 
for attaining their license and typically paid for their own preparation. However, through its 
internal programs and partnerships with several of the external programs, the district was able 
to offer opportunities for aspiring leaders to pursue their credential at low or no cost. 

Some of these preservice training opportunities were distinguished by offering on-the- 
job training through a residency in an NYC DOE school as a central component of their 
programs. According to district data on the preservice training of principals placed into NYC 
DOE schools during the study period, approximately one-third of all principals were trained 
through residency-based programs. Among new principals who received residency-based train- 
ing, approximately 47 percent participated in the district’s internal preservice programs, the 
Leaders in Education Apprenticeship Program (LEAP) and the Assistant Principal’s Institute 


| The Principal Pipeline Initiative was funded by the Wallace Foundation and provided funding to NYC DOE and 
five other districts between 2011 and 2015 to enact multifaceted reforms to improve leadership development and support 
throughout the pipeline. See the Wallace Foundation, 2019 for additional information on the Principal Pipeline Initiative. 
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(API). LEAP is a year-long residency for teacher leaders and APs with Master’s degrees that 
includes a one-year internship at the aspiring principal’s home school (Turnbull et al., 2013, 
2016), whereas API is a preservice training program geared toward individuals aspiring to 
become APs. NYCLA was involved in helping the district to develop the LEAP program, and 
some of its elements were modeled off of NYCLA’s preservice preparation program. 

Outside these district-provided preparation programs, the largest external residency- 
based preservice program was NYCLA’s APP. More information on APP is provided in 
Box 2.1. External residency-based training programs were also offered by New Leaders, NYC 
Bank Street Principals Institute, NYC Fordham AMPEL, and NYC Teachers College Summer 
Principals Academy. As part of its work in the Principal Pipeline Initiative, the NYC DOE 
Office of Leadership began work in 2011 to refine its expectations for school leaders 
and communicate those expectations to external preparation programs serving NYC DOE 
and distributed clear guidance on expectations for aspiring leaders in 2015 (Turnbull et al., 
2016, p. 16). During the 2014-2015 school year, NYC DOE began redirecting its financial 
resources away from external partnership programs and toward its district-run LEAP program 
(Turnbull et al., 2016, p. 19). Despite this shift in emphasis toward internal programs, NYC 
DOE data suggest that the district continued to hire leaders trained by external providers at 
similar rates throughout the study period. 


Leadership Hiring 


Between 2014-2015 and 2016-2017, an average of 132 new principals were hired into NYC 
DOE schools each year. Principal and AP hiring was governed by Chancellor’s Regulation 
C-30 to ensure the hiring process is merit based and equitable. Only candidates who were part 
of the district-wide Principal Candidate Pool were eligible to apply for vacant principal posi- 
tions. To be eligible for the candidate pool, candidates must have had an administrative license 
from the state of New York and meet teaching experience (currently 7 years) and other require- 
ments established by NYC DOE. As part of the Principal Pipeline Initiative, the NYC DOE 
revised its school leader standards during the 2013-2014 school year and aligned the candidate 
pool screening process to those standards. The revised Principal Candidate Pool screening 
involved performance assessments designed to determine whether the candidate meets the 
district’s principal standards (NYC DOE, webpage, undated-b; Turnbull et al., 2013, 2016). 

Once candidates were part of the pool, they had to submit applications for open princi- 
pal vacancies in NYC DOE's districts. The superintendent reviewed applications and selected 
candidates for further evaluation by a larger committee as input to the final principal selection 
(NYC DOE, 2015). Superintendents selected between three and five applicants from the pool, 
who were then interviewed by a school-level committee made up of school staff, parents, and 
a designee of the chancellor. The committee made recommendations to the chancellor, who 
then made a final hiring decision (NYC DOE, 2015). 

When it came to filling AP vacancies, the principal rather than superintendent was the 
hiring authority. Any candidate who met the basic eligibility requirements—which included 
having a New York state administrative license and five or more years of teaching experience— 
could apply. The principal then reviewed applications and made the selection based on 
input from a hiring committee. Although principals had decisionmaking authority when it 
came to filling AP vacancies, staffing decisions related to APs—including termination and 
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reassignment—were governed by contractual agreements between the Board of Education and 
the Council of Supervisors and Administrators of the City of New York. 

The NYC DOE principal and AP hiring processes did not provide special preferences to 
principals who graduated from particular preservice preparation programs. To support high 
rates of placement into NYC DOE schools for their graduates, preparation programs made 
efforts to ensure that their program content reflected the needs of the district and offered 
placement support in the form of networking activities with superintendents and practice 
interviews. 


Coaching and Support for Leaders 


NYCLA began to provide coaching to all first-year principals in NYC DOE in 2005, with 
optional support for some second-year principals (Turnbull, Riley, and MacFarlane, 2013). At 
the time of the TISS program’s development, each new principal in the district was offered 
72 hours of coaching support from NYCLA. Principals at all levels of experience could access 
additional coaching through district support networks, but had to fund that out of their school 
budgets. More detail on NYCLA’s coaching model is provided in Box 2.1. 

Under Chancellor Farifia, the district began to provide coaching internally through the 
new principal support (NPS) program rather than through external partners (Zimmerman, 
2017). The transition began in 2014, and as of 2017 all first-year principal coaching (outside 
the TISS program) was conducted by the district through NPS. The NPS coaching program 
relied on current principals in NYC DOE to serve as Master Principal Coaches for the district’s 
novice principals. NPS offered 72 hours of coaching for each new principal including a mini- 
mum of 36 hours on-site coaching. In exchange for this support, the NPS Master Principal 
Coaches received a payment of $25,000 per year in excess of their compensation as a principal. 
NYC DOE did not have formal supports for all APs, although 44 percent of novice APs sur- 
veyed in 2015 reported that they did receive coaching and mentoring support from the district 
(Turnbull et al., 2016, p. 51). 

Ongoing coaching and support for school leaders was informed by the evaluation process. 
The evaluation process in NYC DOE resulted in a measure of leadership practice that incorpo- 
rated evidence from the principal quality rubric and a measure of student learning. The district 
created a principal practice observation tool that outlined how evaluators were to measure the 
ten indicators in the principal quality rubric (NYC DOE, 2017). Evaluators assessed princi- 
pal practice during supervisory visits by the district superintendent. Superintendents served as 
evaluators for at least one of the required visits but “principal leadership facilitators” also con- 
ducted visits (NYC DOE, 2017). Principals that received low ratings in their evaluations were 
required to create a principal improvement plan for the following school year and additional 
supervisory visits (NYC DOE, 2017). 


CHAPTER THREE 


The Targeted Intensive School Support Program 


In 2013, NYCLA received funding from the U.S. Department of Education through an 
Investing in Innovation (i3) development grant to create the TISS program.' We first provide 
a description of the program, including the five key components of the program, the staff sup- 
porting implementation, and the program participants. We follow with a brief discussion of 
the evidence base that underpinned the development of the program. 


Description of the Program 


The Five Key Components of TISS 

The TISS program was conceived of as an intervention for schools that were identified by 
NYC DOE as struggling and in need of comprehensive intervention. The program aimed 
to prepare a team of two like-minded leaders (or thought partners) that could accelerate the 
change process in these struggling schools and provide the team with robust coaching support. 
TISS was designed around five main program components: (1) preservice teaming and train- 
ing, (2) coplacement of the principal and AP, (3) team-based coaching, (4) extended coaching, 
and (5) needs-focused coaching. In this section, we describe the rationale for each of these key 
components and describe how they compared with the “status quo,” the leadership support the 
district provided to other new principals. We also describe the key activities carried out under 
each of the five program components and how those activities evolved over time in response to 
implementation constraints (described in detail later in this report). Details on the TISS pro- 
gram are summarized in Table 3.1. 


Preservice Teaming and Training 

The TISS program was designed to draw participants from NYCLA'’s existing preservice prep- 
aration programs, including aspiring principals from the APP and APs from the LAP (see 
Box 2.1 for more information on these preservice programs). TISS required an aspiring prin- 
cipal and aspiring AP to team up and receive shared preservice preparation that included 
some team-oriented components. This contrasted with the traditional approach to preservice 
training, in which aspiring principals and APs (including those in NYCLA’s programs) were 
developed as individuals. Aspiring leaders might have engaged with others in their schools of 
employment or residency and might have engaged with other aspiring leaders in a training 
cohort, but would not intentionally have been codeveloped for placement into a school as a 


' This grant program is now called the Education Innovation and Research. 
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team. In addition to the explicit teaming activities that TISS incorporated into the existing preser- 
vice programs, the aligned training curricula used for the APP and LAP programs increased 
the likelihood that TISS partners had similar approaches to leadership, and helped to lay the 
groundwork for effective teaming. 

NYCLA first recruited team-minded individuals into the TISS program from among 
individuals who applied to participate in APP and LAP. Beginning in the fall prior to place- 
ment, NYCLA offered a range of opportunities (e.g., mixers, school visits) for aspiring princi- 
pals and APs from these programs to learn about each other, identify strengths and weaknesses, 
and begin to form partnerships. Program staff provided guidance to help TISS participants 
identify matches between aspiring principals and aspiring APs and approved partnerships 
once identified. Once partnerships were developed, principal and AP teams engaged in formal 
opportunities to build capacity as a team through collaborative work, ranging from occasional 
meetings to a full-semester coresidency at a school. For example, some aspiring principals and 
APs were required to work on a “change project” that focused on real problems of practice from 
the residency school and provided opportunities to develop capacity as a team. These projects 
also provided TISS participants with exposure to the diagnostic tool and coaching model. 

There was variation in the extent of teaming activities across years and across teams 
within a cohort. The first cohort of participants had already started their preservice training 
when the program was funded and initiated, so the preservice teaming and training support 
provided by NYCLA was limited. In the second year, TISS participants were given much more 
information about TISS prior to entering the APP and LAP programs. In addition, with more 
time and formative feedback from the evaluation on the first year of implementation, NYCLA 
greatly enhanced the preservice teaming and training activities for the next two cohorts of 
TISS participants. For example, some of the principal and AP teams in the later cohorts par- 
ticipated in coresidencies. On the other hand, for the later cohorts, NYCLA began to relax its 
requirements for TISS participants to have completed APP, allowing some individuals who 
were just beginning preservice training to be placed into schools. As a result, these individuals 
did not receive the full range of teaming and training exercises specific to TISS (or the entirety 
of their residency-based preservice training). 


Coplacement of the Principal and AP Teams 
As the second key component of the TISS program, TISS aimed to facilitate the coplace- 
ment (or cohiring) of principals and APs into schools as a team. The idea was that these like- 
minded leaders who had been codeveloped as a strong team could enter a school and drive a 
greater degree of change than a principal going into a school alone. Placement had traditionally 
focused on the hiring of principals and APs as separate endeavors, and coplacement of multiple 
leaders at the same time had not been a common practice. To coplace two leaders, schools had 
to have a concurrent vacancy, or add an AP position to the school. Principals may eventually 
have had an opportunity to bring in a like-minded AP, but there was a delay in when principals 
were granted hiring authority, and the principal would have been required to follow the formal 
hiring process, including gathering input from committee members. In theory, there have been 
other district programs (e.g., the Assistant Principal Ambassador program) that could have 
been used to leverage temporary appointments and residencies for coplacement, though these 
opportunities were not used explicitly for the purposes of coplacing leaders into schools. 
According to NYCLA, when the TISS model was developed NYC DOE leadership com- 


mitted to supporting coplacement of teams into schools that were preidentified as being in 
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need of additional support. Such opportunities would have been created through supplemen- 
tal resources for turnaround schools that would create new AP positions concurrent with the 
principal vacancy. Yet between the time the TISS model was developed and implemented, dis- 
trict support for coplacement of TISS principals had waned. Instead TISS principals and APs 
sought positions through NYC DOE's traditional hiring process without any special accom- 
modations. That process, described in Chapter Two, posed challenges for coplacement. To 
overcome these challenges, NYCLA engaged in efforts to inform district superintendents and 
others in the central office about the benefits of the TISS program to facilitate coplacement. 
NYCLA also provided the same support to TISS principals that it has long provided to its APP 
graduates seeking placements as principals—though the support was provided to both leaders 
as a team— including interview preparation, resume building, and networking opportunities 
with district superintendents. 

At the outset, coplacement was established as a requirement for continued support from 
TISS in terms of the three additional coaching components of the program. As TISS entered 
its second year of implementation, barriers to coplacement had become increasingly promi- 
nent, and NYCLA decided to release this restriction. Beginning in the second year, TISS 
principals who were hired into a school as a principal were provided with coaching support 
regardless of whether they were coplaced with their AP partner. 


Team-Based Coaching 
Once TISS principals and APs were hired into schools, NYCLA provided them with team- 
based coaching, which aimed to build cohesion and capacity among the TISS principal and 
AP or other members of the leadership team. A departure from the traditional model of focus- 
ing primarily on one-on-one coaching for the principal under the district’s NPS program and 
NYCLA’s coaching program, 30 percent of TISS coaching time was intended to support the AP 
and/or the broader leadership team within the school as well as the principal. Coaches worked 
with the APs and other leaders individually and with the principal as a team, with the structure 
of the coaching tailored to the needs and preferences of the team. To support TISS principals 
and their leadership teams, TISS coaches received special training on team-based coaching 
strategies and met regularly to discuss the problems of practice facing their leadership teams. 
Because a number of TISS principals entered schools without their AP partner, NYCLA 
had to adapt the focus of team-based coaching to support these principals. In these cases, 
NYCLA coaches first helped TISS principals to identify potential partners among existing 
school leaders. Once leader partner(s) were identified, the coach engaged in team-based coach- 
ing with the principal and the leader partner(s). 


Extended Coaching 

TISS coaching differed from traditional coaching not only in terms of the team focus but in 
the amount and duration of coaching support provided. TISS leadership teams were expected 
to receive 328 total hours of coaching over three years, more than four times the 72 hours that 
has been typically provided to first-year principals in NYC DOE.’ The additional coaching 
provided to TISS schools allowed for more visits by the coach, with many coaches visiting on 
a weekly or biweekly basis in the initial months after placement to provide coaching to the 
principal and the AP partner (or other leaders as the model evolved). Extending the coach- 


2 Non-TISS principals could use school funds to purchase supplemental coaching support. 
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ing over three years ensured ongoing coaching support. NYCLA leadership reported that an 
opportunity to provide three years of coaching to some APP graduates under a prior grant (not 
connected to the TISS program or this study) had found the additional years of coaching to 
be helpful to these principals. 

There was no change in the extended coaching component of TISS over time; it was the 
only program component not to evolve over the implementation period. 


Needs-Focused Coaching 

The final component of the TISS model was to ensure that the coaching support was guided 
by an extensive diagnostic process, whereby TISS leaders identified the greatest needs in the 
school for improvement, set goals for the school and leader development aligned with those 
needs, and aligned coaching to those goals. The aim of the diagnostic process was to help the 
leader(s) entering into a new environment to learn about the school in an intentional way and 
to help them identify leverage for change. New leaders sometimes enter schools without taking 
the time to conduct this deep dive into the school’s context and begin their leadership without 
a clear vision as to where investments should be made in terms of principal effort and leader- 
ship development to address the school’s most pressing needs. In addition, new leader coaching 
often addresses the most pressing need on a given day rather than being driven by an overarch- 
ing set of priorities and school needs guiding the coaching. The diagnostic process helped to 
ensure that leaders were aware of and attending to the school’s most critical needs, and TISS 
coaching was aligned with these identified priorities. 

The diagnostic process consisted of a five-step cycle: (1) review student performance and 
progress data; (2) observe school practice and behaviors; (3) analyze the gathered evidence; 
(4) determine priorities and set short- and long-term goals; and (5) monitor progress toward goals. 
The process was based on the TISS Iceberg model (adapted from Goodman, 2002), whereby stu- 
dent performance and progress data represented the tip of the iceberg, and coaches and school 
leaders were then required to identify the underlying systems and structures and behaviors that 
support these outcomes, as well as the underlying beliefs and assumptions about what drives 
changes in behaviors. TISS coaches encouraged leadership teams to pull together a wide range 
of quantitative and qualitative data to address the diagnostic questions and inform priority/goal 
setting. As soon after placement was possible, the coach and principal were expected to use the 
diagnostic to review student performance and school practices, analyze data, and identify the 
systems and structures working against goals of the school. A data specialist coach was oftentimes 
brought on to help with the data analysis and interpretation. Once the data review process was 
completed, postdiagnostic conferences were held, ideally within a few months of placement into 
the school. These postdiagnostic conferences typically included the principal, the coach, and a 
NYCLA program manager, and in some cases the AP and/or other school leaders and focused 
on using the evidence from the diagnostic to identify goals and create action plans for achieving 
those goals. After the initial data review meeting, the process included a midyear conference to 
review progress and determine whether a change of course was necessary, as well as an end-of- 
year conference designed to assess progress a final time and begin planning for the following year. 

Based on feedback from the initial year of implementation and with additional planning 
time, NYCLA made additional efforts to formalize and emphasize the diagnostic process in 
the second year of implementation. The formal conferences and the involvement of program 
managers were expanded in the second year, and TISS participants were exposed to the diag- 
nostic to a greater degree in the preservice component. 
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Program Staff 

There were many staff supporting the TISS program. As an extension to the coaching NYCLA 
had been providing to NYC DOE for a number of years, the program was able to draw on its 
well-regarded model for leadership coaching and tools to support coaches and principals, and 
a large pool of coaches who had been trained on the coaching model and were familiar with 
the district context. A total of 13 coaches supported TISS over the course of the program, with 
each coach assigned between one and three TISS principals/schools. In addition to support- 
ing TISS participants and schools, some coaches continued to provide coaching to non-TISS 
coaches and engage with other NYCLA programs. 

Additional professional development was necessary to equip coaches with the skills and 
knowledge necessary to provide enhanced coaching under the TISS model. NYCLA estab- 
lished a goal of at least 40 hours of TISS-specific professional development, but records indi- 
cated that coaches received significantly more hours of professional development, ranging from 
55 to 135 hours. Professional development for coaches included formal activities and informal 
opportunities for coaches to discuss problems of practice and share coaching strategies. In 
addition to group professional development sessions, coaches described one-on-one support 
from program managers and other program leadership as valuable professional development, 
allowing them to work through challenges they were facing on the ground. 

To supplement the support provided by NYCLA coaches, TISS leaders had access to 
specialist coaches with expertise in the areas of budgeting, school data analysis, instruction, 
English language learners, special education students, teacher evaluation, and scheduling and 
planning. TISS principals were able to request assistance from specialist coaches at any time, 
and the primary coach helped to facilitate the scheduling of coaching from these specialists. As 
noted previously, the data specialist coach played an important role in assessing school data as 
part of the diagnostic process. 

Outside the coaches, the TISS program was supported by a team of at least nine NYCLA 
staff members who oversaw or supported the program in various ways. These included a pro- 
gram director, the two leads of the preservice programs, three program managers, a project 
manager, a research and evaluation specialist, and the CEO of NYCLA. NYCLA’s Vice Presi- 
dent of Coaching Leadership Services—the person responsible for traditional coaching prior to 
development of TISS—acted as the program’s director. The program director was assisted by 
three program managers with the coaching components, with each program manager assigned 
a set of approximately seven TISS schools to oversee. Program managers were responsible for 
overseeing and supporting coaches, participating in several of the primary diagnostic activities, 
communicating with district superintendents around the progress of TISS leaders and needs 
for additional support, and assisting with various administrative aspects of the program. The 
directors of the preservice programs (APP and LAP) had responsibility for overseeing all of the 
preservice teaming and training and coordinating with those overseeing coaching. The project 
manager provided logistical and administrative support for the program and was responsible 
for collecting and developing program data and documentation. The research and evaluation 
specialist also helped collect and assess program data and provided formative feedback to pro- 
gram leaders. Finally, NYCLA’s then-CEO played a central role in designing the TISS model 
and providing high-level oversight of the program. 
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Table 3.2 
New TISS Participants by Year of Placement into NYC DOE 

2014-2015 2015-2016 2016-2017 Total 
TISS principals hired into a school 8 14 10 32 
TISS APs hired into a school 8 6 1 15 


NOTE: Because of dropping, one principal who left midyear in the 2014-2015 group from analysis, the two 
principals switching schools (one in 2015-2016 and one in 2016-2017), and the two new TISS principals replacing 
former TISS principals, our principal counts and school counts differ. Our school counts in the final analytic 
sample included seven schools that entered the sample in 2014-2015, 15 schools that entered the sample in 
2015-2016, and nine schools that entered the sample in 2016-2017. 


Program Participants 

The implementation and impact findings presented in this report focus on schools rather 
than leaders as the unit of analysis, meaning that a principal had to be placed into a school to 
be included in the analysis. A total of 32 principals and 15 APs from the TISS program were 
placed into NYC DOE schools (Table 3.2). The first group of TISS principals was hired into 
NYC DOE schools in the 2014-2015 academic year and was supported by the TISS program 
through 2016-2017. However, one of these eight principals left the district midyear, whereas 
the other seven continued to be supported by the program. All of the principals in the initial 
year were coplaced with an AP partner. In 2015-2016, additional 14 new TISS principals who 
were hired into NYC DOE schools and one of the principals initially placed in 2014-2015 
moved to a new school. Six of the new principals were coplaced with their AP partner. These 
15 principals and their APs or other leader partners were supported through 2017-2018. In 
2016-2017, an additional 10 TISS principals were placed into schools, and one of the new 
principals placed in the prior year moved to a new school. These 11 principals and their APs or 
other leader partners were supported through 2018-2019. 


The Evidence Base for TISS Program Components 


Researchers have considered the importance of principals on school outcomes since the 1970s 
(Hallinger and Heck, 1996) with rigorous evaluation of how principal training programs con- 
nect to outcomes recently emerging within the empirical literature (Crow and Whiteman, 
2016; Fuller and Hollingworth, 2016; Grissom, Mitani, and Woo, 2019). Several studies docu- 
ment the effectiveness of particular principal preparation programs in preparing novice prin- 
cipals who outperform other new hires in the district (e.g., Corcoran, Schwartz, and Wein- 
stein, 2012; Gates et al., 2019a, 2019b). However, these studies did not attempt to assess the 
effectiveness of the various types of support built into these programs (e.g., preservice training, 
coaching) due to variation in program components across programs and over time (Fuller and 
Hollingworth, 2016; Grissom, Mitani, and Woo, 2019). As a result, there is limited evidence 
on the efficacy of the specific components incorporated into the TISS program. We describe a 
brief overview of the most relevant literature to the TISS program below (much of which was 
established after the development of the TISS program), including a discussion of the evidence 
on (1) leadership coaching, (2) leadership teaming, and (3) strategic data use, drawing from the 
related literature on teacher professional development when appropriate. 


16 A Team-Based Leadership Intervention in New York City Schools 


The Evidence on Leadership Teaming Efforts 

A core element of TISS is the shared training and placement of principals and their leadership 
teams. The evidence on the efficacy of leadership teaming largely relies on principals’ self 
reported outcomes or inferred from the evidence of teacher teaming, where several studies find 
empirical evidence of prominent colleague “peer effects” on student achievement (Jackson and 
Bruegmann, 2009; Opper, 2019; Sun, Loeb, and Grissom, 2017). Based on qualitative data 
from participants in a well-regarded principal development program, Wang et al. (2019) find 
that the ability to build a coalition among existing staff in the school is a critical leadership 
competency, and that the program’s emphasis on coalition building led them to adopt a more 
distributed model of leadership. Several studies also find that cohort models and other forms 
of cotraining of school leaders lead to greater cohesiveness and sense of preparedness (Barnett 
et al. 2000; Huang et al., 2012; Preis et al. 2007). 

There is limited evidence on the efficacy of principal and AP coplacement, and limited 
evidence of district efforts to engage in coplacement. Given the importance of cultural fit on 
the effectiveness of new school leaders (Kowal and Hassel, 2011), the ability to start a posi- 
tion alongside a like-minded AP could be a boon toward establishing a culture conducive for 
success. On the other hand, it is unclear whether the coplacement of two new leaders would 
always reduce cultural barriers, because the pair’s existing relationship could potentially lead 
to issues with cultural fit with other existing school staff. The one relevant study examined 
the Charlotte-Mecklenburg School District’s Strategic Staffing Initiative, which allowed prin- 
cipals electing to transfer to a high-needs school to also bring a five member “leadership team” 
to their new schools and serve as the model for TISS’s teaming model (Pulliam et al., 2012). 
Pulliam et al. (2012) found that while Strategic Staffing Initiative schools generally experi- 
enced growth in student achievement and attendance following transfer of a new principal, 
this growth was not significantly different from that of a selected group of comparison schools. 


The Evidence on Leadership Coaching 

A second core feature of the TISS program was coaching of new school leaders. Leadership 
coaching is a prevalent practice, with a nationally representative survey of principals finding 
that roughly half of surveyed principals reported receiving coaching at some point of their 
career (Wise and Cavazos, 2017), although another national study found that somewhat fewer 
principals receive mentoring support after the first year or two on the job (Johnston, Kaufman, 
and Thompson, 2016). Similar to the teaming literature, studies examining the effectiveness 
of principal coaching typically rely on principal self-reported outcomes, finding that on aver- 
age principals report that access to coaching contributed positively to job performance and 
socioemotional well-being, though these associations appear to be strongly moderated by the 
quality of the principal-coach match (Alsbury and Hackmann, 2006; Crow and Whiteman, 
2016; James-Ward and Salcedo-Potter, 2011; Orr and Orphanos, 2011). A randomized experi- 
ment conducted by Goff et al. (2014) found that access to principal coaching led to improve- 
ments in measures of principal leadership but no effect on teacher instructional quality. To 
our knowledge, no studies explicitly identify the effects of leadership coaching on student 
outcomes. A 2018 meta-analysis conducted by Kraft, Blazar, and Hogan found that teacher 
coaching resulted in positive effects on both instruction and test scores; the authors did not 
find significant differences in the effect of teacher coaching by program features (e.g., contact 
hours, virtual coaching, group training). 
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The Evidence on Strategic Data Use and Priority-Setting by Leaders 

Another core component of TISS was the use of a data-driven diagnostic tool and process to 
identify school needs. The literature suggests that strategic use of data by leadership for priority 
setting and monitoring of progress can help to improve leadership capacity (Bryk et al., 2010; 
Herman et al., 2008; Murphy, 2007). Yet studies using survey data typically find low-grade 
subject-specific correlations between principals’ reported data use and student achievement 
(Faria et al., 2012; Louis et al., 2010). There is some evidence that more tailored data-driven 
interventions are effective. For example, one study found that use of a diagnostic tool as part 
of a coaching program was perceived as being useful (Wang et al., 2019), and the program as a 
whole led to improvements in student achievement in Massachusetts and Pennsylvania (Nun- 
nery et al., 2011; Nunnery, Ross, and Yen, 2010). Van Geel et al. (2016) found significant posi- 
tive achievement effects of an intervention in the Netherlands designed to train school leaders 
to implement data-based decisionmaking. 


CHAPTER FOUR 


Study Approach 


NYCLA contracted with RAND to conduct independent implementation and impact evalu- 
ations of the TISS program between 2014 and 2020. The evaluation was designed around the 
following research questions: 


1. How was the TISS program implemented? 
a. ‘To what degree was the program implemented as planned (i.e., with fidelity)? 
b. Were there adaptations made to the program? 
c. What were the barriers and facilitators to implementation of TISS? 
2. What were the effects of TISS on schools with participating leaders relative to schools 
with new residency-trained principals who were not supported by TISS? 
a. Did TISS participation lead to improvements in student performance on standard- 
ized mathematics and ELA assessments? 
b. Did TISS participation lead to improvements in student attendance? 
c. Did TISS participation lead to improvements in school climate? 
d. Did TISS participation lead to improvements in principal retention? 


In this section, we first briefly describe our approach to assessing implementation and effect. 
We conclude with a brief overview of limitations to the study. Appendix A provides additional 
details on the data and methods used for the study. 


Research Design 


Although the “gold-standard” for impact investigations is the randomized control trial, the 
intentional selection of participants from the APP program and the district’s placement pro- 
cess for new principals made it infeasible to randomize principals to the TISS program and 
to schools. Yet we could not simply compare outcomes for TISS principals and schools to 
other principals and schools in NYC DOE, because there were likely to be systematic differ- 
ences between schools with TISS principals and other schools in the district (Shadish, Cook, 
and Campbell, 2002). In particular, the TISS program called for principals to be put into 
the highest-needs schools, so we needed to compare their outcomes to outcomes for similarly 
struggling schools. 

We used a three-step process to identify our comparison group. The first step was to 
choose a pool of principals comparable to TISS principals across two characteristics: years of 
experience as a principal and type of preservice training received. Because all of the TISS prin- 
cipals were new, first-year principals, we selected comparison schools that also had first-year 
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principals. In addition, because TISS program participants were all drawn from NYCLA’s 
residency-based APP program, we needed to find a way to disentangle the effects of TISS from 
the effects of the APP program. To do this, we limited the pool of comparison schools to those 
with principals who had completed a residency-based principal preparation program, includ- 
ing other APP principals who did not participate in TISS, graduates from the district’s LEAP 
and API programs, and graduates from other external residency-based preservice training pro- 
grams. Although not a perfect comparison—with residency-based programs likely varying in 
design and effectiveness—this group of other new principals from residency-based programs 
served as the best possible comparison group available for the analysis. We also required that 
TISS and comparison principals were placed in the school (with or without coplacement of an 
AP) by November 15 of an academic year, and they needed to remain in the school through 
May 15 of that first academic year to allow sufficient time for treatment to occur. This restric- 
tion led us to drop one TISS principal from the sample who left the district midyear. 

The second step was to eliminate schools that did not have sufficient data to be included 
in the analysis. The school must have been open for at least one year at the time of placement to 
ensure that preprogram outcomes (e.g., test scores, student attendance, etc.) were available for 
the quasiexperimental analysis process. This restriction eliminated an additional three TISS 
principals/schools from the analysis, resulting in a total of 28 treatment schools across the three 
years of placement (seven entering the sample in 2014-2015, 12 in 2015-2016, and nine in 
2016-2017). There were 54 eligible comparison schools with new, residency-trained principals. 

Descriptive statistics for the eligible TISS schools and the unweighted comparison schools 
are provided in Table 4.1. Although TISS principals were hired through the regular process 
rather than being explicitly placed into schools that were predesignated as “high needs,” data 
suggest that schools with TISS principals did have characteristics suggesting higher than aver- 
age risk. TISS schools had test scores that were below the district average and lower than 
other comparison schools with new, residency-trained principals. In addition, the popula- 
tions of TISS schools were primarily made up of students from minority racial/ethnic groups 
(51 percent Hispanic and 37 percent Black), and the majority of students in TISS schools were 
living in poverty (83 percent). A greater proportion of students at TISS schools relative to com- 
parison schools fell into these groups. 

After limiting our pool of comparison principals to residency-trained principals with sim- 
ilar experience, and limiting the sample of schools to those with sufficient data, we needed to 
account for differences between the schools with TISS principals and the other new, residency- 
trained principals. We used a propensity score weighting approach to ensure that the compari- 
son schools were similar to treatment schools in terms of their prior year student performance 
on standardized assessments. To check that the weighting was successful in creating a com- 
parison group that was closely matched to the treatment group, we inspected the standard- 
ized mean differences of baseline scores on ELA and math assessments. As anticipated, after 
weighting, both baseline ELA and mathematics scores have nearly equivalent means in the 
TISS and weighted comparison schools, and demographic and socioeconomic characteristics 
of the two groups also looked similar (Table 4.1). 

The small remaining differences between students in TISS principal schools and weighted 
comparison schools were accounted for through our analytic approach. We assessed the effect 
of the TISS program by comparing the school-average outcomes of treated and weighted com- 
parison schools. We used a generalized linear model, weighted by the estimated propensity 
score, regressing the student outcomes on whether they had been led by a TISS principal that 
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pen we Statistics for the Study Sample in the Year Prior to Principal Placement 
Unweighted Weighted 
TISS Schools Comparison Comparison 
(n=28) Group (n=54) Group (n=54) 
Black 0.37 0.33 0.32 
Hispanic 0.51 0.44 0.47 
English language learner 0.14 0.15 0.21 
Students with disabilities 0.20 0.21 0.21 
Poverty? 0.83 0.76 0.77 
Standardized ELA score (baseline) -0.30 -0.14 -0.30 
Standardized math score (baseline) -0.24 -0.12 -0.24 


*Poverty is defined based on student family income relative to locally calculated poverty thresholds 
(see NYC IBO, 2015). 


year and included covariates in our regression models to obtain estimates of the treatment 
effect (Bang and Robins, 2005, Hullsiek and Louis, 2002). Covariates included math and 
ELA assessment scores for the prior year, school grade configuration (e.g., high school, middle 
school, K—8 school), as well as other school characteristics including the proportion of students 
with disabilities, the proportion of English language learners, the proportion of African Ameri- 
can students, the proportion of Hispanic students, the proportion of economically disadvan- 
taged students, chronic absenteeism, and number of years that the principal has worked in the 
district. All models use robust standard errors and include cohort fixed effects. 

One potential limitation of these analyses is that they did not control for differences in 
baseline trends for TISS schools and comparison schools prior to receiving a new principal. It 
is possible student outcomes in treatment schools were systematically increasing (or decreas- 
ing) prior to the start of the study in ways that differ from comparison schools. To test the 
sensitivity of our findings to baseline trends, we also conducted comparative interrupted time 
series (CITS) analyses which examine changes in outcomes over time for students in schools 
led by TISS principals and students in schools not led by TISS principals and explicitly account 
for any differences in baseline student outcome trends. These models are discussed in greater 


detail in Appendix A. 


Data and Measures 


Implementation Measures 

RAND worked collaboratively with NYCLA to determine the measures and data sources used 
to assess implementation fidelity (Table 4.2). NYCLA requested that fidelity analysis primarily 
focus on the postplacement components of the program, so we were limited to a single mea- 
sure of fidelity for the preservice component, completion of the APP program by the TISS 
principal. We characterized the fidelity of the coplacement component according to whether 
the TISS principal was placed with the AP she or he teamed with during preservice training. 
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Table 4.2 


Implementation Data and Measures 


Program Component 


Implementation of teaming, 
preservice training 


Implementation of placement 


Fidelity to team-based coaching 


Fidelity to extended coaching 


Fidelity to needs-focused 
coaching 


Perceived barriers and 
facilitators 


Data Source(s) 


Program tracking data 


Program tracking data 


Program tracking data (professional 
development attendance sheets) 


Coaching logs 


Coaching logs 


Diagnostic documentation 


Feedback document completed 
by principal and coach 


Feedback document completed 
by principal and coach 


Interviews with principals, coaches, 
NYCLA leadership, superintendents 


Measure(s) 


# of principals completing APP 


# of principals coplaced 


Cumulative hours of TISS-specific 
training for coach by end of year 


Percent of yearly coaching hours 
that included others on team 


Total cumulative hours of coaching 
at end of Y1, Y2, Y3 


Completed the diagnostic 


Diagnostic data/evidence used 
to inform goals 


Coaching aligned with goals 


Open-ended reporting of 
barriers and facilitators 


We examined two measures of fidelity to the team-based coaching component: coaching time 
devoted to team coaching across the three years and hours of TISS-specific professional devel- 
opment received by TISS coaches. To examine fidelity to extended coaching, we assessed the 
total number of hours of coaching in each year following placement. Our examination of fidel- 
ity to needs-focused coaching examined completion of the diagnostic tool/assessment designed 
to assess the root causes of issues within the school, use of the diagnostic assessment to deter- 
mine goals, and alignment of coaching with goals. For a school to have implemented a key 
component with fidelity, all measures for a given component must have satisfied the required 
minimum thresholds for fidelity. To assess fidelity at the program level, NYCLA determined 
that 75 percent of all schools with TISS principals must be implementing each component 
with fidelity. 

To assess stakeholder experiences with the implementation of the TISS program and bar- 
riers and facilitators to implementation, we conducted semistructured interviews with a range 
of stakeholders involved with implementation during the first two years of the grant. A total 
of 39 interviews were conducted, including 11 interviews with TISS principals, 14 interviews 
with TISS coaches, 11 interviews with TISS program leadership, and 3 interviews with super- 
intendents. More detail on the interview sample and content of the interviews is provided in 
Appendix A. We analyzed interview data to identify qualitative themes related to how the pro- 
gram was implemented, barriers and facilitators to implementation, and stakeholder percep- 
tions of the program. 


Impact Measures 

We examined the school-level effect of TISS across several different outcomes (Table 4.3). Our 
primary (confirmatory) outcomes of interest were student performance on standardized math 
and ELA assessments. For students in grades 3 through 8, these assessments included the NY 
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Table 4.3 

Impact Data and Measures 

Outcome Data Source(s) Measure(s) 

Student achievement Student-level district NY State Math and ELA 
administrative data Test (grades 3-8), NY State High School 

Regents Exams (high schools) 

Student attendance Student-level district Chronic absenteeism (absent 10% 
administrative data of days enrolled or more) 

School climate Publicly available NYC School Survey (parent, 
school-level data teacher, and student forms) 

Principal retention Staff member-level district Remained as NYC DOE 
administrative data principal, staff 


State English Language Arts Test and the NY State Math Test. For high school students, these 
assessments included the NY State High School Regents Exams in ELA and Algebra I. We 
also conducted exploratory analysis of the extent to which TISS reduced chronic absenteeism, 
defined as being absent more than 10 percent of the days of enrollment within an academic 
year. School climate was measured based on the NYC School Surveys, which were adminis- 
tered annually to students, parents, and instructional staff. These surveys measured six dimen- 
sions of school climate, including the extent to which (i) instruction is customized, inclusive, 
motivating, and aligned to the Common Core; (ii) the school establishes a culture where stu- 
dents feel safe, supported, and challenged; (iii) teachers are committed to school success and 
improvement; (iv) school leadership is effective; (v) the school fosters strong family and com- 
munity ties; and (vi) school staff, parents, students, and administrators trust and value each 
other. Finally, we explored the extent to which TISS drove increases in principal retention, 
which we defined as staying in the principalship in NYC DOE. 

The confirmatory analysis focused on school-level average outcomes three years after the 
principal’s initial placement. We also explored school-level average outcomes after one and two 
years and examined outcomes across the three cohorts of participants to gain a more nuanced 
understanding of outcomes and explore whether effects changed over time. Findings from 
these exploratory analyses are detailed in the Appendix B. 


Study Limitations 


There were a number of limitations to our study approach. First, with only 28 TISS schools 
and relatively few comparison schools with residency-trained principals, the statistical power 
of our analysis was limited. Small sample sizes restricted the number of covariates that could 
be included in matching and analysis, and we faced challenges with the stability of parameter 
estimates. The small sample size also limited our ability to examine the relationship between 
effects and key aspects of implementation or school characteristics. 

Our approach to developing a comparison group may not have accounted for all of the 
selection issues. For comparison group principals who also received preservice training from 
NYCLA’s APP program, there were likely unobserved differences between principals who did 
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and did not select into the TISS program that we were not fully able to account for. In com- 
paring TISS principals (trained through the APP program) to principals who came from other 
residency-based programs, we made the assumption that there were no differences in the effec- 
tiveness of these different programs. To the degree that NYCLA’s APP was more or less effec- 
tive than these other preservice preparation programs, these differences on the effect of APP 
versus other residency-based training programs might have been included in our estimates to 
the effect of TISS. 

Beyond these issues with selecting a good comparison group, we were limited in our abil- 
ity to describe a clear intervention and compare it to a stable counterfactual. The key compo- 
nents of the TISS program and program activities changed substantially over time, meaning 
that different cohorts received varying treatments. In addition, there was variation within 
cohorts in the intervention received, with varying intensity in terms of preservice teaming 
experiences and varying postplacement teaming experiences depending on whether the princi- 
pals were coplaced. We had limited data in some cases to capture this variation due to decisions 
at the outset of the study to focus fidelity analysis on the coaching-related components and to 
focus the qualitative data collection in the first two years of the program. In addition to vari- 
ability in how TISS was implemented, leadership preparation and support for all principals in 
the district were changing during the study period, such as NYC DOE moving first-year prin- 
cipal coaching in-house to its NPS program rather than using NYCLA for this coaching. We 
had anticipated having robust data on the coaching received by comparison principals from 
NYCLA records and NYCLA survey data, but given that most of these comparison principals 
were trained through NPS we had very limited data on the coaching and support received in 
the program. 

We also faced limitations with our ability to examine several outcome measures. NYC 
DOE implemented a revised version of its school survey beginning in the 2014-2015 academic 
year. Although we included the previous school survey scales as baseline measures in all analy- 
ses, it should be noted that the new school survey did not measure the same constructs as the 
previous school survey. In addition, response rates for these surveys varied across schools, and 
it is not possible to determine the extent to which survey respondents are representative of the 
school community as a whole. We were unable to examine effects on leadership quality ratings 
because these quality ratings were not collected on an annual basis for all principals, with wide 
variation in the timing of measurement that was systematically tied to principal quality. 

Finally, there are limitations with the generalizability of our findings. All of the TISS 
schools were in NYC DOE, a district that is unique in many ways, outperformed other dis- 


tricts in the state, and already had a robust leadership support system in place when TISS was 
created. The findings are therefore difficult to generalize beyond NYC DOE. 


CHAPTER FIVE 


Findings on Implementation of the TISS Program 


Our implementation analysis examined the degree to which the TISS program was imple- 
mented as planned (fidelity), adaptations to the program over time, and barriers and facilitators 
to implementation. 


Implementation Fidelity 


The program-wide fidelity results for the implementation of the TISS program indicate that 
two of the program’s key components were implemented as planned, whereas the other three 
were not (Table 5.1). A total of 23 schools (82 percent of the sample) received TISS principals 
who had completed NYCLA’s preservice training, and the same number of schools imple- 
mented the needs-focused component with fidelity. The team-based coaching component was 
implemented with fidelity at 19 schools (68 percent), whereas coplacement and extended coach- 
ing were implemented with fidelity for just 14 schools (50 percent). We provide a more detailed 
description of our implementation findings for each component in the following sections. 


Preservice Teaming and Training 
NYCLA was relatively successful in drawing most TISS principals from the APP, and this 
helped to ensure that the principals could benefit from the preservice components of the TISS 


Table 5.1 
Percentage of Schools Implementing TISS Program Component with Fidelity 


Year 1 TISS Schools | Year 2 TISS Schools | Year 3 TISS Schools 


(2014-2015) (2015-2016) (2016-2017) Total 
APP completion 67% | s5% | tam | 
Coplacement 50% 11% 50% 
Team-based coaching 71% 58% | 7% | cam 
Extended coaching 71% 42% 44% 50% 
Needs-focused coaching 71% | 
Total schools 7 


NOTE: Numbers represent percentages of schools in the final analytic sample implementing with fidelity. Cells 
highlighted in green indicate that the component was implemented with fidelity at the program level, meaning 
that at least 75 percent of schools are implementing the component with fidelity. Cells in red indicate that the 
component was not implemented with fidelity at the program level. 
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program. After NYCLA relaxed the requirement that TISS principals had to have completed 
the APP, we found that five TISS principals were placed into schools without completing the 
preservice training, including four who had committed to participation in the APP but left 
early to enter the principalship, and one who was pulled from the LAP. However, completion 
of preservice training by principals was just one aspect of TISS’s preservice training require- 
ments, so this single measure is limited in assessing fidelity and does not capture the variabil- 
ity in the matching and team capacity building across cohorts and principals within cohorts. 
Although we were aware of substantial variation in the participation of TISS participants in 
various teaming activities across years and within cohorts, we did not have consistent data on 
preservice activities to characterize this variation in preservice experiences. 

Because the preservice teaming components largely focused on building capacity between 
specific principal and AP pairs (as opposed to building transferrable teaming skills), the value 
of the preservice teaming activities for the 50 percent of principals who were not placed with 
an AP may have been greatly diminished. NYCLA’s shift to place a greater emphasis on build- 
ing broad team-building skills in the final cohort rather than focusing exclusively on preser- 
vice teaming with a specific AP may have increased the likelihood that the preservice teaming 
components offered value to all TISS participants regardless of whether they were coplaced. 


Coplacement into NYC DOE Schools 

NYCLA encountered substantial challenges placing TISS participants into NYC DOE schools. 
The program initially aimed to place 50 teams of TISS principals and APs across the three 
cohorts, yet only 32 TISS principals were ultimately placed in NYC DOE schools. After rec- 
ognizing that the district would not be granting hiring preferences to TISS leader teams and 
loosening its restriction that TISS principals be coplaced with an AP after the first cohort, 
fidelity to coplacement declined substantially. By year two, only 50 percent of TISS principals 
were coplaced with an APP, and in the third year of implementation only one TISS principal 
was coplaced with an AP. Across the full sample, only half of all TISS schools were led by a 
TISS principal and AP who were coplaced. 

There were a number of factors potentially contributing to coplacement issues according 
to NYCLA staff and NYC DOE administrators. According to NYCLA staff, NYC DOE had 
initially committed to helping place TISS participants into struggling schools, but did not 
end up providing NYCLA with any accommodations in the hiring process. TISS principals 
entered the regular pipeline for hiring, and a limited number of schools likely had simultane- 
ous openings for principals and APs in NYC DOE, leaving TISS leaders with a potentially 
limited pool of schools to choose from. And although there was some flexibility for superinten- 
dents (and eventually principals) to carve out new positions in the school budget, this required 
strong buy-in for TISS among superintendents, which the program did not necessarily have. In 
some cases, there were issues with the teams themselves that contributed to coplacement chal- 
lenges. For example, some partnerships also faltered in the interview process, with principals 
and APs not able to articulate their strength as a team, or superintendents liking one partner 
and not the other. Finally, some of the principal and AP pairs ended up deciding to part ways 
prior to interviews or after placement based on incompatibility or interpersonal differences. 


Team-Based Coaching 
All of the TISS coaches received the targeted amount of professional development on team- 
based coaching, so this aspect of team-based coaching did not hinder fidelity of implementa- 


Findings on Implementation of the TISS Program 27 


Figure 5.1 
Percentage of Total Coaching Hours Delivered to the Broader Leadership Team 
Three Years After Placement 
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NOTE: Data come from the hourly coaching logs and records on coplacement. Each bar represents a 
TISS school. Values represent the percentage of total coaching hours that included others on the 
leadership team over three years of TISS support. The solid black line represents the stated target 
NYCLA set for team-based coaching. The dotted line represents the fidelity threshold. 


tion. However, the program ran into challenges ensuring that all schools received the amount 
of team-based coaching required. The recommended amount of coaching that was specified 
for inclusion of others on the leadership team was 30 percent, with a threshold of 20 percent 
as a minimum for implementation with fidelity. The percentage of total coaching hours 
that included others from the leadership teams in TISS schools ranged from 5 to 58 percent 
(Figure 5.1). Of the 28 schools in the sample, 19 met the minimum threshold for fidelity 
and 12 exceed the 30 percent recommended. As shown in Table 5.1, fidelity was particu- 
larly low for schools that entered the sample in Year 2. This was when the program began 
to place a number of TISS principals without their AP partner, and NYCLA did not yet 
have a plan in place for how TISS principals might engage in team-based coaching if not 
placed with their partner during the second year. However, NYCLA staff had reenvisioned 
team-based coaching for the principals hired into schools in the third year of the TISS 
program, and the rebound in fidelity for the group of principals entering in the third year 
of implementation suggests that team-based coaching could be implemented in schools 
where coplacement had not occurred. 


Extended Coaching 

We assessed fidelity of implementation to the extended coaching component by measuring the 
total cumulative coaching hours received by the school at the end of each of the first three years 
after placement of the TISS principal. As described in Table 3.1, the targets for yearly cumu- 
lative hours were 164 by the end of Year 1, 292 by the end of Year 2, and 328 by the end 
of Year 3. Minimum fidelity thresholds were set at 70 percent of target hours, meaning that 
TISS was implemented with fidelity when coaching hours exceeded 114 hours in Year 1, 204 
hours in Year 2, and 229 hours in Year 3. The total number of coaching hours over three years 
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Figure 5.2 
Total Hours of Coaching Three Years After Placement 
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line represents the fidelity threshold. 


of TISS support varied widely by school, from 98 to 477 hours (Figure 5.2). After the third year 
of TISS support, 17 of the 28 study schools had leaders who had exceeded the minimum fidel- 
ity threshold of at least 229 total coaching hours over three years, and most of these schools 
had also exceeded the target of 328 total hours of coaching. However, four of the schools that 
received more than 229 years of cumulative coaching missed a fidelity threshold for hours of 
coaching in prior years, resulting in just 50 percent of all TISS schools at which the extended 
coaching portion of TISS was implemented with high fidelity (Table 5.1). 


Needs-Focused Coaching 

Overall, the needs-focused coaching of the TISS program—completion of a diagnostic process 
and use of this process for goal setting and coaching—was implemented with high fidelity, 
with 82 percent meeting implementation targets. In the cases where needs-focused coaching 
was not implemented with fidelity, it was due to the principal not completing the diagnostic 
at all, rather than completing the diagnostic and then not using it for the purposes of goal set- 
ting and coaching. Table 5.1 indicates that engagement with the diagnostic varied from year to 
year, with particularly high rates of fidelity for schools entering the sample in the second year. 
Interviews with program staff and participants provided further evidence of improvements in 
implementation after the first year. Several of the principals placed in the first year did not 
remember completing the diagnostic, and those who did use the diagnostic reported it to have 
played little or no role in determining priorities and guiding coaching. In addition, none of 
the coaches who were interviewed in the first year of implementation described the diagnos- 
tic as a key feature of the TISS model. In contrast, most of the stakeholders we interviewed 
from Cohort 2 identified needs-based coaching as a key feature of the TISS model and could 
articulate the value of the diagnostic in enhancing coaching and improving leadership capac- 
ity. Interview findings suggested that the higher rates of fidelity for later cohorts were driven 
by changes NYCLA made to emphasize and formalize this component of the model. 
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Adaptation of the TISS Program over Time 


As noted in the initial description of the TISS program and Table 3.1, there were a number 
of changes that occurred to major program components over time. Regarding the preservice 
component, NYCLA had additional time to enhance teaming and training activities for later 
cohorts, and to establish expectations and build buy-in for the program, and enhanced oppor- 
tunities for partnership-building. For example, NYCLA required aspiring TISS principals 
from later cohorts to complete extensive projects with their AP partners, with some partners 
completing these projects in shared residencies at schools. These structured projects exposed 
participants to the diagnostic and provided them with opportunities to work with teams and 
mentor principals to address real leadership challenges in their residency schools. In contrast, 
team-building activities in the first year of TISS implementation consisted of occasional meet- 
ings. Aspiring principals in later cohorts also had more exposure to TISS coaches during the 
preservice components, and according to program leadership, coaches, and principals, this 
early exposure facilitated a smoother transition to the coaching components of TISS. 

When NYCLA realized that many of the principals were not going to be coplaced, TISS 
leaders modified some aspects of the preservice component to better support principals placed 
without an AP partner. During preservice training for the final cohort of TISS participants, 
the discussions of teaming focused to a greater degree on general teaming with all leaders in 
a school. These changes to how NYCLA thought about teaming also permeated the team- 
based coaching component; coaches worked with principals who were not coplaced with an 
AP to identify other leaders they might partner with during the first year of placement, and 
then these members of the existing leadership team were eventually brought into coaching 
discussions. 

Finally, NYCLA modified the diagnostic process and communication around the process 
for later cohorts. Key changes were enhanced messaging about the importance of the diagnos- 
tic process to coaches and principals, greater use of the diagnostic in preservice training, and a 
more formal meeting structure built around the diagnostic during the first year of placement. 
Both principals and coaches reported that the more structured process helped to keep them on 
track with assessing the data, setting goals, and aligning coaching to those goals, and the pres- 
ence of the program manager at check-in meetings was reported as underlining the importance 
of the process. 


Facilitators and Barriers to Implementation 


Conversations with coaches, principals, NYCLA leadership, and district leadership highlighted 
a range of factors within the context of NYCLA and NYC DOE that acted as barriers and 
facilitators to implementation with fidelity. We identified the following three main facilitators 
to the implementation of TISS: 


¢ Buy-in of coaches and staff: When principals believed in the value of TISS and its approach 
to leadership support, TISS activities were viewed as valuable and supportive, and prin- 
cipals were willing to devote time and effort to the program’s activities. Coaches played 
an important role in encouraging and supporting principal engagement, so their buy-in 
to the model was also important. Our interviews suggested high levels of buy-in among 
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coaches, with coaches acting as strong advocates of the program. In addition, the active 
engagement of coaches in helping to refine the TISS model was important, because many 
of the strategies for effective team-based coaching and needs-focused coaching were devel- 
oped on the ground by coaches as they worked with TISS leadership teams. 

° Support of program staff: Coaches spoke often of the support that NYCLA staff provided 
and the important role of that support in facilitating implementation. Program man- 
agers helped coaches to use the diagnostic and acted as a thought partner (or coach) to 
the coaches when they were struggling. They also helped to interface with the district to 
market the program and gather feedback on participants. Program leaders were perceived 
as being available as needed and regularly solicited feedback from principals, coaches, 
and district administrators in an effort to assess implementation and make midcourse 
improvements. The collaboration of NYCLA leadership overseeing the preservice train- 
ing components and the coaching components was also helpful in ensuring a cohesive 
TISS program. 

° Flexibility of NYCLA to adapt the program: NYCLA’s flexibility allowed the program 
to support TISS participants in widely varying circumstances, and the willingness to 
adapt also helped to address early issues observed with implementation. For example, 
the changes to the preservice and diagnostic components were well-received according to 
interviewees and may have driven the higher rates of fidelity on the needs-focused coach- 
ing component. The similar rates of fidelity to team-based coaching for principals who 
were not coplaced suggest that TISS participants were able to receive team-based coach- 
ing despite the challenges with coplacement. 


There were also several aspects of the context in NYCLA and NYC DOE that acted as barriers 
to successful implementation, including the following: 


¢ Limited understanding of the program: Interviews with principals and coaches indicated 
wide variation in understanding about what TISS was among program participants in 
the early years of implementation. Several principals reported that they were not given 
advanced notice of what they had “gotten themselves into,” and that this had affected 
their ability and willingness to engage in the program’s activities. NYCLA made some 
adjustments for later cohorts to expand professional development and preservice training 
activities around TISS, and coaches reported a greatly improved understanding of TISS 
in later years. Yet a few principals from later cohorts reported they were ill-informed 
about the program and could have benefited from clearer early communication and the 
opportunity to “opt in” to TISS. The fact that some participants exited the preservice 
training components early may have limited their understanding of the program. Inter- 
viewees also suggested that better communication with key stakeholders in the district 
(e.g., district superintendents) from the central office may have been useful in building 
understanding and buy-in. 

° Limited buy-in among district stakeholders: There was a lack of buy-in among several 
groups of district stakeholders. Although NYCLA leaders and district administrators 
both described strong support from district leadership and buy-in for TISS when the 
program was created, this buy-in diminished over time, and the district provided limited 
support for coplacement. There were also some challenges with buy-in among existing 
school staff. Some coaches and principals who were coplaced reported that other APs and 
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school staff lacked trust and were unhappy with what they viewed as a special role for 
the AP that was coplaced with the TISS principal. In schools where coplacement was not 
implemented, some TISS principals faced challenges engaging the existing school staff in 
teaming and team-based coaching efforts due to issues with trust and buy-in around the 
value of teaming and coaching. 

Limited buy-in among some TISS participants: Although the buy-in of coaches and some 
TISS participants helped to facilitate the TISS program at some schools, other TISS par- 
ticipants were less enthusiastic about the program. For example, several TISS principals 
reported regrets for having participated. These TISS participants were likely to engage 
less often with coaches and engage to a lesser degree with other aspects of the program. 
Limited placement and coplacement: Placement rates for TISS principals were lower than 
expected, and only half of the principals were placed with their leader partner. When 
coplacement failed, it became a significant barrier to the ability to implement and experi- 
ence effects from other components of TISS. First, the value of shared preservice training 
was only realized when principal and AP partners were coplaced. Principals were likely 
frustrated when placed without APs they had invested substantial time building relation- 
ships with, and this may have reduced their buy-in for other aspects of TISS. Team-based 
coaching had to be completely reenvisioned to accommodate principals who were placed 
alone, and these principals and their coaches had to spend time identifying new partners 
and building buy-in for team-based coaching. Coaches and principals reported that this 
led to delays in when the additional hours of coaching could be used and lower percent- 
ages of those hours being used for team-based coaching. There were a number of factors 
potentially contributing to coplacement issues according to NYCLA staff and NYC DOE 
administrators. First, a limited number of schools happened to have simultaneous open- 
ings for principals and APs in NYC DOE, so TISS leaders may have had a more limited 
pool of schools to choose from. And although there was some flexibility for superinten- 
dents (and eventually principals) to carve out new positions in the school budget, this 
required strong buy-in for TISS, which was not the case among most superintendents. 
Some partnerships also faltered in the interview process, with leader partners not able to 
articulate their strength as a team, or superintendents liking one partner and not the other. 
Finally, some of the principal and AP pairs ended up deciding to part ways prior to inter- 
views or after placement based on incompatibility or interpersonal differences. 

Time constraints for new principals: According to both principals and coaches, new prin- 
cipals faced an overwhelming number of responsibilities in their first year. TISS com- 
ponents required additional time from principals, APs, and other leader partners on top 
of the traditional leadership responsibilities. Some principals and coaches reported chal- 
lenges finding time to get coaching in, and several principals reported it tough to sit 
down and focus during coaching sessions knowing how much else had to be done around 
the school. When additional leaders were brought into coaching the time constraints 
multiplied, and it was challenging to get leaders into coaching together. The diagnostic 
component was viewed as particularly time-intensive. Principals and coaches reported 
that time constraints were especially problematic for principals who were placed late in 
the summer or after the start of the school year. At least one other study of leadership 
coaching suggests that the time demands of these programs can be a significant challenge 


(Wang et al., 2019). 


CHAPTER SIX 


Findings on Effects of the TISS Program 


In this section, we first present results on standardized achievement scores, capturing the extent 
to which TISS schools did or did not drive improvements in student achievement relative to 
similar schools three years after placement of a new residency-trained principal. We then pre- 
sent exploratory results that capture the extent to which TISS led to improved outcomes after 
three years in terms of chronic absenteeism, school climate, and principal retention relative to 
outcomes for schools with other residency-trained new principals. 


Effects of TISS on Student Achievement 


Figure 6.1 shows our estimate of the effect of the TISS program on student outcomes, includ- 
ing performance on ELA and math assessments three years after principal placement. The stu- 
dent achievement estimates represent the covariate-adjusted difference between TISS schools 
and weighted comparison schools in average standardized ELA and mathematics achievement 
scores. These estimates provide information about the extent to which principals who partici- 
pated in the TISS program saw improved outcomes in the academic achievement of their stu- 
dents when compared with schools that had other new principals from residency-based train- 
ing programs. A negative difference indicates that on average TISS schools underperformed 
relative to the comparison schools in terms of average achievement scores three years after 
principal placement. A positive difference indicates that TISS schools performed better than 
comparison schools on a particular outcome. 

Our analysis had limited statistical power to identify meaningful effects. In both mathe- 
matics and ELA, the estimates were negative, but not statistically significantly distinguish- 
able from zero. First- and second-year results are presented in Appendix B and show similar 
patterns. 


Effects of TISS on Attendance Outcomes 


Our analysis of student outcomes also included an attendance measure, which we measured 
as the reduction in the proportion of students who are chronically absent (i.e., absent for 
more than 10 percent of enrolled days). We compared the average chronic absenteeism rate for 
schools with TISS principals to the average outcome for our weighted comparison group. Find- 
ings suggest that there were statistically significant differences (at the 5 percent level) between 
TISS schools and the weighted comparison group in terms of chronic absenteeism three years 
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Figure 6.1 
Third-Year Effects of TISS on Student Achievement Relative to Comparison Schools 
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SOURCE: NYC DOE Administrative Data (AY 2015-2016, 2016-2017, 2017-2018, 2018-2019). 
NOTE: Neither estimate is statistically significant at the 5 percent level. Estimates are based on 28 treatment 
schools and 54 comparison schools. 


after placement. Specifically, we find the rate of chronic absenteeism was 4.8 percentage points 
greater in schools receiving TISS principals as compared with the comparison schools. More 
details on this analysis are presented in Appendix A. 


Effects of TISS on School Climate Outcomes 


Figure 6.2 presents our estimates of the effect of TISS on school climate outcomes three years 
after the placement of a new principal. All school climate scales are reported on a percent posi- 
tive metric. For example, a score of 90 percent can be interpreted as meaning that 90 percent 
of stakeholders had positive perceptions of school climate. Estimates represent the covariate- 
adjusted. difference between the average school climate score or schools with TISS principals 
and the average outcome for weighted comparison schools with new residency-trained princi- 
pals who had not participated in TISS. 

Overall, our estimates of the effect of TISS on school climate were negative but were not 
statistically significant. One- and two-year results are presented in Appendix B and indicate 
similar patterns. 


Effects of TISS on Principal Retention Outcomes 


Finally, we estimated the effect of TISS on principal retention three years after initial place- 
ment. As mentioned previously, we defined retention as remaining as a principal in NYC 
DOE for a full academic year. To be included in the sample, both the TISS principals and 
the comparison principals had to be retained throughout the first year, so by design we have 
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Figure 6.2 
Third-Year Effects of TISS on School Climate Relative to Comparison Schools 
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100 percent one-year retention. Estimates of retention after three years were positive represent- 
ing an increase in retention of approximately 6 percent, but this difference was not statistically 
significant. 


Effects of TISS by Program Implementation 


As discussed in the previous chapter, there was substantial evolution over time in the imple- 
mentation of the TISS program, and the program was not implemented with fidelity for many 
of the schools where TISS principals were placed. It is useful, therefore, to examine whether 
those schools where TISS was implemented with greater fidelity were more likely to experience 
positive effects (or less likely to experience negative effects). We conducted exploratory analysis 
to examine schools where principals were coplaced and schools with fidelity across a greater 
number of key components (at least four of the five components). It is important to note, how- 
ever, that because principals could self-select into their level of participation, these analyses 
cannot be interpreted causally. In addition, we were underpowered to detect differences in 
effects with such small sample sizes. 

We examined relationships between implementation and outcomes by focusing on two 
school-level measures: whether a principal was coplaced with their AP partner and whether 
TISS was implemented with fidelity for at least four of the five components. Findings from 
these exploratory analyses revealed no statistically or practically significant relationships 
between outcomes and exposure to program features. In other words, these findings suggest 
that schools where TISS was implemented with fidelity according to these measures fared no 
better or worse than schools where TISS was not implemented with fidelity (see Appendix B 
for details). 


CHAPTER SEVEN 


Discussion 


In designing the TISS program, NYCLA aimed to build an innovative approach to develop- 
ing and supporting leadership for struggling schools. Drawing on evidence that highlighted 
the importance of having a cohesive leadership team, the TISS program offered an innovative 
team-based approach to preparation and support for leaders, as opposed to the individual focus 
of most principal preparation and early career support programs. In addition, TISS aimed 
to provide robust coaching to participants, with more than four times the number of coach- 
ing hours offered to other new principals in NYC DOE and the use of a diagnostic process 
that aligned coaching with school priorities through deep inquiry and collection of data and 
evidence. 

However, there were substantial challenges with the implementation of the TISS pro- 
gram, with only two of the program’s five components implemented with fidelity. The failure 
of coplacement—a central feature of the TISS model—was a serious barrier to the program’s 
implementation and success, because the effectiveness of the other team-based components of 
the program including the preservice activities and the team-based coaching hinged on fidelity 
to coplacement. Many factors impeded coplacement including leadership transitions and other 
leadership reforms in the district that diminished buy-in, limited assistance from the district 
with TISS leader placement, structural constraints on cohiring (i.e., lack of open slots, school 
budget limitations), and issues with cohesion among some teams. Many of these barriers to 
coplacement were outside NYCLA’s control. 

It became clear in the initial years of implementation that coplacement would become a 
major challenge and would have ripple effects on all other aspects of the program, and NYCLA 
was flexible and responsive in adapting its program to reenvision its teaming concept. How- 
ever, these changes may have been confusing to program participants and district leadership, 
whose buy-in to the intensive program was critical. The program was also designed for strug- 
gling schools with particularly strong needs for support, and although TISS principals were 
placed in schools that on average had lower achievement, schools that some TISS principals 
ended up in may not have been struggling to the degree that required more comprehensive 
support. 

Given these circumstances, it is not surprising that we did not find any evidence of 
improved student and school outcomes related to the TISS program. Findings suggested no 
statistically significant differences between TISS schools and comparison schools with other 
new residency-trained principals on math or ELA achievement, school culture, and princi- 
pal retention outcomes, whereas TISS schools underperformed relative to comparable schools 
in reducing chronic absenteeism. Across a range of outcomes, time periods, and comparison 
groups, we found no evidence of statistically significant positive outcomes. 
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Beyond implementation challenges, there may be other factors that contributed to our 
findings on the program’s effect. NYC DOE is not a typical district; it has a strong record of 
support around leadership development (including a long history of work with NYCLA), and 
evidence on student achievement indicates that the district has been outperforming similar 
districts in the state. As a result, comparison schools were also receiving some coaching support 
(as all principals in NYC DOE receive), and the contrasts in support between the treatment 
and comparison schools may not have been large. In another district, where the “status quo” 
in terms of leadership support was less developed or nonexistent, we might have expected more 
positive effects from leadership support interventions. 

Our analytic approach was also limited in that it required comparison of TISS principals 
to other residency-based principals in the district to disentangle the effect of TISS from the 
effects of preservice preparation. If the APP participants who selected into TISS were different 
than other APP graduates, and if other residency-based preservice preparation programs were 
more effective than APP at preparing leaders, then comparing TISS participants to these other 
residency-trained principals may have understated the effects of the TISS program. 

Whenever possible, leadership interventions should be designed in ways that align with 
standard hiring and staffing practices: 


¢ When leadership interventions are offered by external organizations, close coordination with 
and strong buy-in from districts are important to successful implementation, but may not be 
sufficient to overcoming shifting priorities: NYCLA has long worked with NYC DOE on 
leadership interventions and has built strong relationships with district leadership and 
staff. Understanding the importance of district buy-in, NYCLA worked collaboratively 
with NYC DOE to design the TISS program. Despite these efforts to ensure district 
buy-in and coordination, changes in NYC DOE leadership and major reforms to prin- 
cipal pipelines in the district were taking place alongside the roll-out of TISS, leading to 
major unanticipated shifts in district buy-in around the TISS program and involvement 
in the program’s implementation. 

¢ Flexibility and adaptation in the face of challenges with program implementation may be 
necessary to sustain the program but pose evaluation challenges: The K-12 education con- 
text, particularly in large urban districts, is dynamic and organizations implementing 
multiyear programs to support school leaders in this environment may need to adapt pro- 
gram elements in response to changing circumstances and feedback from participants. In 
adapting, organizations face trade-offs between fidelity to the original approach on the 
one hand and feasibility and responsiveness to the needs of the district and target popula- 
tion. Although such adaptations allow the program to be sustained, they make it difficult 
to assess the model as originally envisioned. 

° Leadership support programs that hinge on placement decisions are risky: Many different 
factors are at play in determining which schools principals end up in and what the make- 
up of their leadership team looks like, and most of these decisions are outside the control 
of organizations that partner with school districts around leadership. As a result, interven- 
tions that hinge on the placement and retention of specific principals and APs in specific 
schools are risky, especially when the intervention requires significant departures from 
standard hiring and staffing practices. Wherever possible, leadership interventions should 
be designed in a way that is aligned with standard hiring and staffing practices. For 
example, the adapted TISS model that focused on team building with existing staff in a 
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school was more flexible in its ability to serve principals regardless of setting and required 
less upfront investment, and may have been less vulnerable to diminished support from 
the district. A qualitative study of another leadership development and coaching program 
suggests that developing a leader’s capacity to build coalitions among existing staff is a 
critical competency that should be fostered among leaders (Wang et al., 2019), suggesting 
that the adapted version of TISS that did not require coplacement may have been equally 
or more valuable. 

° Tools and resources can support coaching and may also be useful to those with more limited 
coaching support: The diagnostic tool and process that were incorporated into the TISS 
model was one of the few components implemented with fidelity, and many of the partic- 
ipants found the diagnostic process to be useful. And at least one other study found diag- 
nostic to be a useful tool for leadership support (Wang et al., 2019). NYCLA has incorpo- 
rated this diagnostic into other leadership support efforts outside NYC DOE. Although 
we cannot determine whether this component conclusively benefited TISS participants 
and schools relative to comparison schools, the positive implementation evidence suggests 
it might be beneficial to continue to incorporate the diagnostic into leadership support 
programs offered by NYCLA and consider replicating the process in leadership programs 
offered by other organizations and districts. 


In conclusion, the TISS program offered an innovative approach to early career leadership 
support that was designed to improve coaching and reconceptualize early career support as a 
team-based endeavor. Although evidence suggests that TISS did not lead to better outcomes 
for students and schools relative to traditional leadership support for other new residency- 
trained principals in NYC DOE, the findings offer important lessons learned to districts and 
leadership support organizations about the implementation of team-based support interven- 
tions. In addition, NYCLA leadership reports that TISS introduced some enhancements to 
their broader coaching model (e.g., use of the diagnostic) that are now being incorporated into 
support offered by NYCLA to districts and leadership support providers across the country. 


APPENDIX A 


Methods for the Report 


Implementation Study Approach 


Implementation Study Data Sources 

Given limited resources for implementation analysis, we relied largely on data collected by 
NYCLA for the study. This included coaching hours for all coached principals, annual coach- 
ing goals and progress toward goals, documentation from the diagnostic process, professional 
development data, and survey data from principals who receive coaching from NYCLA. We 
supplemented these data with interviews with a range of stakeholders. We describe each of 
these data sources below. 


Interviews with Stakeholders 

RAND conducted interviews toward the end of Years 1 and 2 of implementation (2015 and 
2016) between March and July. Individuals interviewed for the study included TISS princi- 
pals, TISS coaches, NYCLA staff; and NYC DOE district superintendents. Table A.1 pre- 
sents the number of interviews by interviewee type and interview year. All TISS principals in 
cohorts 1 and 2 who were placed in a school and received a full year of TISS coaching were 
invited to participate in an interview at the end of the first year of coaching. In addition, all 
NYCLA staff and NYCLA coaches who were involved with TISS were asked to participate 
in interviews. Finally, we asked NYCLA staff to identify a sample of district superintendents 
and other district staff who had experiences with the TISS program and TISS participants and 
could speak to the implementation of the program. NYCLA identified six district superinten- 
dents and a high-level district administrator whom RAND contacted for an interview. 

The Year 1 interviews with NYCLA staff and coaches were conducted in person, while 
all other interviews were conducted by phone. Interviewees were asked about a range of topics 
that aligned with the research questions, including (1) understanding of the TISS program; (2) 
implementation of recruitment and training activities, coplacement, and coaching (including 
team-based, extended, and needs-based components); (3) outcomes of participation in TISS; 
and (4) overall perceptions and lessons learned regarding TISS. Protocols for our various stake- 
holder groups are provided in Appendix C. 

Each interview included an interviewer and a note-taker, and detailed notes were taken 
during each interview. Notes were deidentified and stored securely to protect potentially sensi- 
tive data. 


Coaching Time Log Data 
To account for the amount of coaching for compensation purposes, coaches were required to 
log the time they spent coaching into Tenrox, the NYCLA timekeeping system. In addition 
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Table A.1 
Number of Interviews Conducted by Year and Type of Interviewee 


2015 2016 
First-year principals 4 7 
Coaches 4 10 
NYCLA staff 4 7 
District superintendents 0 3 
Total 12 27 


to entering the time spent coaching, coaches were asked to indicate whether that coaching 
was provided solely to the principal, or whether other members of the school leadership team 
participated. 

To account for the amount of coaching a principal received, we aggregated hours of coach- 
ing provided between July 1 and June 30 of a given year. We calculated total hours of coaching 
in a year and total hours of team-involved coaching in a year for each TISS principal in each 
year of participation. 


NYCLA Program Tracking Data 

Two types of program documentation were used to determine whether needs-based coach- 
ing activities were implemented as planned. First, the diagnostic process was documented 
using program manager notes at four key check-in points in the diagnostic process, includ- 
ing the initial diagnostic data dive, the postdiagnostic debrief, the midyear review meeting, 
and the end-of-year conference. This documentation was used to provide evidence that the 
diagnostic process was used at a particular TISS school. Second, as part of the coaching pro- 
cess, principals and coaches were required to document up to three goals for the academic 
year for the school and/or principal. These goals were documented in a “Goals/Progress” 
tracking sheet that was regularly updated. For each goal, the principal and coach specified 
the evidence used to identify that goal. At the end of the academic year, the principal and 
coach also reported whether (1) the goal had been “completely met”; (2) there was “signifi- 
cant progress”; (3) there was “some progress”; or (4) there was “no progress.” These data were 
used to assess whether goals were informed by the evidence and whether these goals were 
met through coaching. 


Program Documentation 

We also used program documentation to describe various aspects of the TISS model. 
Documentation that we reviewed included the initial program proposal; training materials 
for principals and coaches, a description of the pre-service change project and accompany- 
ing rubric, the diagnostic tool, slides from a presentation given at the annual i3 confer- 
ence, documentation related to advisory committee meetings, a needs assessment report 
NYCLA developed to inform the design and implementation of TISS, and survey analysis 
reports. 
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Methods 
Fidelity Analysis 
Fidelity measures were developed for each school in the sample, with a focus on the three coach- 
ing components of TISS. NYCLA and RAND worked collaboratively to identify measures that 
were used to assess fidelity, and NYCLA developed the thresholds for high implementation. 
Table A.2 presents information on the fidelity metrics for each component, and we 
describe our processes for assessing fidelity below. First, fidelity was measured at the school 
level for the five key components for all schools in the impact analysis sample. Then, the 
percentage of schools that were implementing with high fidelity was calculated to determine 
program-wide fidelity. For program-wide fidelity, NYCLA considered a component to have 
been implemented with high fidelity if at least 75 percent of schools in the sample were high 
implementers of that component in a given year. 


Fidelity to Teaming and Shared Training 
NYCLA requested that fidelity analysis primarily focus on the postplacement components of 
the program, so we were limited to a single measure of fidelity for the preservice component, 


completion of the APP program by the TISS principal. 


Fidelity to Coplacement 
We assessed fidelity to coplacement according to a single measure, a 0/1 indicator of whether 
the TISS principal was placed with the AP she or he teamed with during preservice training. 


Fidelity to Team-Based Coaching 

To assess whether TISS was meeting expectations for implementation regarding team-based 
coaching, NYCLA identified two fidelity metrics: (1) access to coaches who were adequately 
trained in TISS and team-based coaching and (2) evidence that the coach was providing 
coaching to others on the leadership team beyond the principal. NYCLA identified the ideal 
amount of TISS-focused training for coaches to be 40 cumulative hours, and set 30 cumula- 
tive hours as a minimum threshold for high implementation. Internal tracking of professional 
development participation was used to calculate the cumulative hours of TISS-focused train- 
ing. Although the fidelity metric for access to an adequately trained coach was measured on 
an annual basis, coaches did not need to receive training each year; they needed only to have 
obtained 30 cumulative hours of training over time. 

Regarding the second metric, the amount of coaching provided to others on the leader- 
ship team, NYCLA established an ideal target of 30 percent of total coaching hours going to 
others aside from the principal. The minimum threshold for high implementation was set at 
20 percent of total coaching hours. The fidelity metric was measured annually, determined by 
the coaching provided only in that year. 

A school was considered high implementing in the area of team-based coaching if the 
coach and principal exceeded thresholds for both of these metrics; if they failed to meet either 
or both of the thresholds, then the school was not determined to be high implementing. 


Fidelity to Extended Coaching 

To assess whether TISS was meeting expectations for implementation regarding extended 
coaching, we examined total cumulative hours of coaching at the end of each year. Data on 
total hours of coaching were drawn from the coaching time records. TISS participants were 
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Table A.2 
Fidelity Metrics 


School-Level Fidelity 


TISS Component Fidelity Metric Threshold Data Source 
Teaming andshared __ # of principals completing No threshold identified Program tracking data 
training APP 
Coplacement # of principals coplaced No threshold identified Program tracking data 
Team-based Cumulative hours of TISS- At least 30 hours (75% of ideal, Program tracking 
coaching specific training for coach 40 hours) data (professional 

by end of year development 


attendance sheets) 


Percent of yearly coaching At least 20% (2/3 of ideal, 30%) Coaching logs 
hours that included others 


on team 
Extended coaching Total cumulative hours 114 in Year 1 Coaching logs 
of coaching 190 in Year 2 
215 in Year 3 
(70% of planned hours) 
Needs-focused Used the diagnostic Some evidence of diagnostic Diagnostic 
coaching use by principal documentation 
Evidence used to inform Coach and principal cite data/ Goals/progress 
priorities/goals evidence as informing at least worksheet 
one of the goals set 
Coaching aligned Coach and principal report that Goals/progress 
with goals at least one goal was completely = worksheet 
met, or that significant progress 
made? 


2 This measure is not completely aligned with the fidelity metric originally designed for the program. Ideally, 

we would have data on principal perceptions of alignment, and NYCLA included a question on the survey that 
directly asked principals about this. However, due to incomplete survey data and the need to assess fidelity for all 
schools/principals, we had to substitute this measure. 


ideally supposed to receive 164 hours of coaching by the end of their first year in the program, 
292 hours of coaching by the end of Year 2, and 328 hours of coaching by the end of Year 3. 
NYCLA established thresholds for high implementation at 70 percent of the expected coach- 
ing hours (resulting in thresholds of 114, 204, and 229, respectively). We considered princi- 
pals who met the thresholds for each of the three years as high implementing in the area of 
extended coaching. 


Fidelity to Needs-Focused Coaching 
To assess whether TISS was meeting expectations for implementation regarding needs-focused 
coaching, NYCLA identified three fidelity metrics. The first fidelity metric tracked whether 
the diagnostic tool/process was used at least once at a school. The second metric assessed 
whether TISS participants identified priority areas, or goals, according to data/evidence. 
Finally, the third fidelity metric determined whether coaching had been aligned with the iden- 
tified priorities. 

Initially, the plan was to use survey data from NYCLA’s end-of-year coachee survey to 
address these fidelity metrics. Participants who responded “yes” on survey questions related 
to the three metrics would be identified as high implementing schools. However, given low 
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response rates, these data could not be used to assess fidelity. Instead, we used NYCLA pro- 
gram documentation related to the diagnostic process and coaching to determine fidelity. 

Individuals who had diagnostic-related documentation were identified as satisfying the 
first needs-focused metric. The other two fidelity metrics were drawn from the goals/progress 
sheet. When principals set goals, they were required to also describe the evidence that sup- 
ported the selection of that particular goal. If a principal reported that at least one of their 
identified goals was supported by data, then the second metric for high implementation was 
met. Finally, if a principal reported that at least one of their goals had been “met” or had 
“significant progress,” then the third metric for high implementation was met. This metric 
does not exactly capture the intended measure of “coaching aligned with goals,” though for 
goals to be satisfied, we might assume that coaching was aligned. Only when principals had 
accounted for all of three components of needs-focused coaching was their school considered 
high implementing. 


Analysis of Interview Data 

Although analysis of program documentation and tracking data were essential for our research 
question on fidelity, analysis of interview data helped to address the other implementation 
questions. A coding scheme was developed and refined by the analysis team based on our 
research questions, and notes were coded by two researchers. The coded data were analyzed 
to identify themes around resources (¢.g., staffing, training); implementation of various study 
components, including teaming and preplacement training, placement into a school, coach- 
ing, and use of the diagnostic; perceptions of implementation success; perceived facilitators and 
barriers; and recommended changes. 


Impact Analysis Approach 


In this section, we provide additional details about the quasiexperimental methods used in the 
impact analyses. First, we describe the propensity score weighting algorithm. We then describe 
the statistical models used to estimate TISS program effects, including our primary model 
specifications and the CITS models used for the sensitivity analyses. Additional details are 
also provided on the outcome variables, treatment and comparison definitions, and baseline 
equivalence of the treatment and comparison groups. We conclude by presenting results for 
the confirmatory analyses that examined outcomes at three years following the placement of 
treatment (TISS) principals in schools as well as exploratory analyses that examined outcome 
at one- and two-year time points, analyses by cohort, and the CITS analyses. 


Propensity Score Methods 

When comparing students in TISS schools to counterparts throughout their district, selection 
bias poses a serious threat to the validity of inferences because schools and leaders participating 
in the treatment may be systematically different than those who are not (Shadish, Cook, and 
Campbell, 2002). To mitigate this threat, we use propensity score methods (¢.g., Stuart, 2007) 
to create a comparison group that is as similar as possible to the treatment group based on a set 
of observable baseline characteristics. For these analyses, we first limited the comparison pool 
to include only schools receiving a new, residency-trained principals in each academic year. 
Subsequently, we used the following variables to create a matched comparison group: 
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¢ Placement school’s prior year ELA scores 

¢ Placement school’s prior year mathematics scores 

¢ Placement school’s grade levels served (with dummy coded variables representing elemen- 
tary, middle, and high schools, as well as K-8 and 6-12 schools) 


Although in general, it is desirable to use a wide range of baseline variables to estimate 
propensity scores, because of the limited sample sizes in these analyses, we focus our propensity 
score estimation on a small set of variables. However, we include a wider range of variables in 
our impact models (see below) and in our appraisal of baseline covariate balance. 

We use the R package twang (Ridgeway et al., 2014) to create weights for individuals in 
the comparison group so that, on average, they closely matched the treatment group. Twang 
uses generalized boosted regression models to estimate propensity scores. Although propensity 
scores are typically estimated using parametric models that are linear in the unknown parameters 
(e.g., logistic regression) to calculate the likelihood, or the propensity, that each school is treated 
given their observable characteristics, generalized boosted regression model is a nonparametric 
piecewise constant model that is iteratively estimated using regression trees (Setodji et al., 2017). 
The advantage of using a generalized boosted regression model is that it does not assume linear- 
ity, automatically accommodates interactions between covariates, and provides stable propensity 
score estimates even where few observations are available (Setodji et al., 2017). The twang soft- 
ware allows for the estimation of propensity scores and for appraising covariate balance in treat- 
ment and comparison groups. We estimated propensity scores separately within each cohort and 
then pooled the cohorts into a single file for analysis, following the design illustrated in Table A.3. 


Analysis Methods 

After generating the propensity score weights, the treatment effect of the TISS program can 
be obtained by comparing the outcomes of treated and weighted comparison schools. The 
treatment effect is estimated using a generalized least squares model, weighted by the esti- 
mated propensity score, and using sandwich-estimated standard errors to account for cluster- 
ing within principal.' We use the following so-called “doubly-robust” model—which employs 


both propensity score weights and covariate adjustment—to obtain estimates of the treatment 
effect (Bang and Robins, 2005; Hullsiek and Louis, 2002): 


Ise= Bot ByTic+ Bo Icco +A Ae 6, esc (1) 


Yee is the outcome for school s in cohort c. 7; indicates the school’s treatment assignment (0 for 
comparison and 1 for treatment). X,, is a vector of school level characteristics, defined below. 
Yeo is a measure of the outcome variable at baseline. 6, is a cohort-specific fixed effect, and e,. 
is a random effect with mean zero and variance 02. The effect of TISS on school outcomes 
is tested by testing the null hypothesis so that 8, =0. This model is used as the basis for our 
confirmatory contrasts, using data from the third posttreatment year for all cohorts. 

The following preintervention covariates were included in X, for the models examining 
student outcomes and principal retention: 


1 Tfa principal stays in a school for one full academic year and then moves to another school for another full academic year, 


both schools would be included in our analysis. This creates the possibility of clustering of schools within principals. 
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Table A.3 
Matching Variables and Outcome Variables by Cohort 

Cohort 1 Cohort 2 Cohort 3 
2013-2014 Match on: 


e ELA scores 
¢ Math scores 
¢ Grade configuration 


2014-2015 Match on: 
e ELA scores 
¢ Math scores 
¢ Grade configuration 


2015-2016 Match on: 
e ELA scores 
e Math scores 
e Grade configuration 


2016-2017 Third-Year Outcomes 
2017-2018 Third-Year Outcomes 


2018-2019 Third-Year Outcomes 


¢ School average state test scores (mathematics and ELA) 
¢ Proportion of students who were African American 

¢ Proportion of students who were Hispanic 

¢ Proportion of students with disabilities 

¢ Proportion of students with English learner status 

¢ Proportion of students living in poverty 

¢ Proportion of chronically absent students 

° School grade-level range 

¢ Total years of experience in the principalship 


For school climate variables, we included all of these covariates in X, and also included prein- 
tervention measures of the six school climate scales.” 


Exploratory Analyses of First- and Second-Year Effects and Cohort-Specific Effects 
We conducted additional exploratory analyses to estimate the effect of TISS after the first and 
second years of the study. For these analyses, we used models identical to specification detailed 
in Equation (1), including the same propensity score weights and covariates. 

Additionally, we conducted the third-year analyses separately for each cohort. These anal- 
yses used a slightly modified version of the model in Equation (1): 


Vsc= Bo aH B, i + B, sc + XA oe esc (2) 


2. For the first cohort, these six scales were not administered in the baseline year. Therefore, for that cohort, three different 
baseline scales were used. 
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We estimate this model separately for each cohort (c= 1,2,3) to obtain cohort-specific effects. 
Veo Lies Xoo Veco» and e,. are as defined above. 


SO Ss 


Exploratory Analyses of Associations Between Third-Year Effects and 

Fidelity of Implementation 

We conducted additional exploratory analyses to estimate the extent to which impact estimates 
are associated with the TISS program itself and the extent to which these estimates can be 
attributed to low program exposure or uptake among certain TISS participants. We focused 
our fidelity analyses on two specific indicators: an indicator of coplacement (given the impor- 
tance of this particular component for the overall success of the program and an indicator of 
overall high fidelity of implementation (four to five areas implemented with fidelity). Fidelity 
analyses are based on a modification of the models used for one-, two-, and three-year impact 
analyses (Equation 1): 


In= Bot BT, + By I0+ BZ AXA 0 (3) 


where all variables are as defined previously, y,, is the outcome for school s in time t after place- 
ment and Z, is a measure of fidelity. The effect of participating in TISS but reporting no fidel- 
ity of implementation or coplacement is tested based on f,. The difference between this effect 
and the effect of participating in TISS programming with complete fidelity of implementation 
is tested based on 3. To obtain an estimate of the program had it been fully implemented for 
all participants, we use the linear composite of the estimated parameters f, + 3. 


Comparative Interrupted Time Series Analysis Methods 

As a sensitivity test, we also conducted CITS analyses using the propensity-weighted schools 
for the achievement outcomes. We used the propensity score weighted linear model given in 
Equation (1) as our main specification because we did not have sufficient historical data for 
all variables to run the CITS models. This allowed us to use a simplified modeling framework 
with a common statistical model for all analyses. The CITS analysis controls for baseline trends 
for two baseline years and estimates the extent to which differences before and after the imple- 
mentation of an intervention can be attributed to the intervention rather than to other factors: 


ise Os o.+ aT, 


ste 


Ut Cas (4) 


In this model, y,,. is the outcome at time ¢ for school s in cohort c, which importantly now 
includes two preplacement years in the regression. y,,. is a function of a school-specific fixed 
effect 6, a time-specific fixed effect 6, and a 3X1 vector of treatment indicators 7,,. that 
indicates how long school s in cohort c had been treated in year ¢. TT\,, 
interaction terms that provide estimates of year-specific effects. Note that, unlike Equations 


then represents three 


(1) and (2), these models do not employ baseline demographics as covariates. We also include 
a school-specific linear time trend 7,’t, which controls for differential baseline trends. We also 
ran traditional difference-in-difference specifications that did not include the school-specific 
linear time trends and found similar results. Finally, the residual errors are given by e,,.. which 
are clustered at the school level in the analysis. 

Our key parameters of interest are contained in 7, which provide estimates of how much 
scores changed over time in TISS schools after placement beyond the change that non-TISS 
school experience in receiving a new residency-trained principal. 
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Sample 

Details on Sample Exclusion and Inclusion Criteria 

In this section, we provide some additional details on how the treatment and comparison 
conditions were defined. As described in the main report, we define a school as treated if a 
TISS principal is assigned to that school for one full academic year, which we define as having 
started in a school by November 15 of a given academic year and remaining in the school until 
May 15 of the same academic year. If a principal vacates a position prior to the completion of 
one academic year, that school will not be considered as treated and will not be eligible to be 
identified as a comparison school.’ If, however, a principal is assigned to a school and remains the 
principal of record for one full academic year, that school will be considered treated, even if 
the principal vacates that position after that first year is complete. If the principal is reassigned 
to another NYC DOE school, then that new school would be considered treated if (1) the 
principal remains the principal of record for one full academic year and (2) outcomes would 
be available for analysis in the study time frame. In the event that a TISS principal vacates a 
school after the completion of an academic year, and another TISS principal is placed into that 
same school, that school is considered treated and included (as a single case) in our analyses as 
a member of the initial treatment cohort. 

To have a consistent criteria for both treatment and comparison schools, a school is only 
eligible to be identified as a comparison school if (1) the school’s principal entered the princi- 
palship in the same cohort as the treated principals and has served in the school for at least one 
full academic year; (2) the school’s principal completed a residency-based leadership training 
program; and (3) that school is not treated at any point during the time frame of the study. 
For example, a school that is a potential control unit for Cohort 1 (because that school had a 
new non-IISS principal for Year 1 would be excluded from the pool of potential controls if 
that school was assigned a TISS principal in Year 2 or 3. If a principal leaves a school before 
the end of the first school year and leaves the district altogether, that school would be removed 
from the analyses. 


Baseline Equivalence of Treatment and Comparison Groups 

NYC DOE provided data on all public schools for the academic years 2013-2014 through 
2017-2018. After restricting the data set to eligible treatment and comparison schools, there 
were 28 treatment schools and 54 comparison schools. 

The characteristics of these schools are presented in Table A.4. All of the variables presented 
were measured at baseline, that is, prior to the appointment of a TISS principal. For schools 
in the Cohort 1 sample, this baseline reflects data from 2013 to 2014 academic year. For 
schools in the Cohort 2 sample, this baseline reflects data from 2014 to 2015 academic year. 
For schools in Cohort 2 sample, this baseline reflects data from 2015 to 2016 academic year. 

In Table A.4, it is evident that, prior to adjusting for propensity score, characteristics of 
the treatment and comparison group schools had substantial differences. The columns labeled 
as SMD display the standardized mean differences between the two groups. What Works 
Clearinghouse criteria define standardized mean differences greater than 0.25 standard devia- 
tions as evidence that treatment and control groups are not equivalent. Many of the SMDs in 
the first two columns of Table A.4 are quite large, and, in general, the TISS schools are lower 


3 There was one TISS principal in Cohort 3 that did not meet the criteria for treatment. 
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achieving, have higher minority enrollment, have higher proportions of economically disad- 
vantaged students, and have higher rates of chronic absenteeism than the comparison schools. 

Table A.4 also shows the improvement in baseline equivalence after applying propen- 
sity score weights to the comparison pool. Many of the variables with large differences prior 
to weighting show dramatic improvements in balance. For example, when compared with 
the unweighted sample, TISS principals were entering schools with much lower average ELA 


Table A.4 
Descriptive Statistics for Three Cohorts in the Year Prior to Principal Placement 


Before Propensity Score Weighting After Propensity Score Weighting 
Other Other 
(n=54) TISS (n=28) SMD (n=54) TISS (n=28) SMD 
Black 0.33 0.37 0.16 0.32 0.37 0.11 
(0.25) (0.23) (0.24) (0.23) 
Hispanic 0.43 0.51 0.34 0.47 0.51 0.09 
(0.24) (0.23) (0.23) (0.23) 
English language learner 0.15 0.14 -0.04 0.21 0.14 -0.17 
(0.17) (0.16) (0.26) (0.16) 
Students with disabilities 0.21 0.20 -0.14 0.21 0.20 -0.08 
(0.08) (0.06) (0.09) (0.06) 
Chronic absenteeism 0.31 0.35 0.27 0.32 0.35 0.14 
(0.18) (0.13) (0.14) (0.13) 
Poverty 0.76 0.83 0.49* 0.77 0.83 0.21 
(0.18) (0.12) (0.18) (0.12) 
ELA (baseline) -0.14 —0.30 -0.34 —0.30 -0.30 0.00 
(0.50) (0.40) (0.55) (0.40) 
Math (baseline) -0.12 -0.24 -0.25 -0.24 -0.24 0.00 
(0.52) (0.42) (0.50) (0.42) 
Collaboration 82.15 83.05 0.1 82.27 83.05 0.04 
(9.05) (9.10) (8.83) (9.10) 
Effective leadership 78.74 80.14 0.13 78.57 80.14 0.07 
(11.75) (10.17) (12.33) (10.17) 
Rigorous instruction 83.41 83.52 0.01 83.23 83.52 0.02 
(8.06) (7.32) (8.27) (7.32) 
Family ties 84.47 81.81 -0.35 84.08 81.81 -0.14 
(6.63) (8.91) (6.41) (8.91) 
Supportive environment 80.65 78.24 -0.26 80.94 78.24 -0.15 
(9.53) (8.72) (9.15) (8.72) 
Trust 86.82 87.62 0.11 86.88 87.62 0.05 
(8.23) (5.97) (8.43) (5.97) 


NOTE: Prior to 2014-2015, NYC DOE administered a different version of the school survey. For this reason, 
information from the Cohort 1 schools is not included in the baseline estimates for collaboration, effective 
leadership, rigorous instruction, family ties, supportive environment, or trust. For these characteristics, other 
N=34, TISS N=21. Poverty is defined based on student family income relative to locally calculated poverty 
thresholds (see NYC IBO, 2015). 


Abbreviation: SMD, standardized mean difference. * p<.05. 
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and mathematics scores than non-IISS new principals; with weighting they are now nearly 
equivalent. For all variables, the standardized mean differences between treatment and control 
groups are less than 0.25 standard deviations, making them equivalent as per What Works 
Clearinghouse standards. Using this threshold of standardized mean differences less than 0.25, 
Table A.4 shows that, after weighting, the treatment and control groups are equivalent on all 
baseline observed covariates. 


Measures 

Academic Achievement and Chronic Absenteeism 

We used NY State assessment scores in ELA and mathematics as measures of achievement. For 
students in grades 3—8, these assessments include the NY State English Language Arts Test 
and the NY State math Test. This data was provided to RAND by the NYC DOE for all stu- 
dents enrolled in NYC public schools for the school years between 2012-2013 and 2018-2019. 

For elementary and middle schools, school scores were created in three steps. First, we 
standardized individual student scores into z-scores within grade level and subject using city- 
wide means and standard deviations, which we calculated from student-level administrative 
data for all students enrolled in NYC public schools in a given school year. Second, we aggre- 
gated these standardized scores by grade and school, so that each school had a set of school- 
level scores for each grade level (e.g., an elementary school would have a school-level third-, 
fourth-, and fifth-grade scores). Third, we formed weighted averages (weighted by the number 
of students in each grade) across all grade levels within a school to form a single achieve- 
ment score for each subject. For elementary and middle schools, students in grades 3-8 were 
included in the school averages. 

High school students were assessed using the NY State High School Regents Exams in 
ELAs and Algebra I. In high schools, students often take the Algebra I Regents exams more 
than one time during the academic year. Because of this, we took a student’s score as their 
highest score achieved during the year. For ELA, taking the exam multiple times was less 
common, and so, after consultation with individuals with knowledge of the district testing 
policy, we took scores on the June administration as the score of record. School scores were 
then created by standardizing individual student scores within district. For high school exams, 
we standardized individual student scores into z-scores separately for each Regents Exam using 
city-wide means and standard deviations, which we calculated from student-level administra- 
tive data for all students enrolled in NYC public schools in a given school year. 

For chronic absenteeism, district administrative data contain information about both the 
number of days enrolled and the number of days absent. Based on this information, we con- 
structed a variable showing the proportion of enrolled days that a student was absent. Consis- 
tent with district policy, students missing more than 10 percent of enrolled days were flagged 
as chronically absent. School-level chronic absenteeism was constructed by aggregating to the 
school level across all students in the school in a given year. 

Key descriptive statistics for these three student outcome measures are provided in 


Table A.5. 


School Climate 
We used six scales from the NYC School Survey as outcome measures of school climate. The 


NYC School Survey has been administered annually to all parents, teachers, and students in 
grades 6-12 since 2014-2015. The six scales we used are described in Table A.6. 
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Table A.5 
Descriptive Statistics for the Third-Year Student 
Outcomes in Treatment and Comparison Schools 


N Mean SD 
Math 82 -0.250 0.488 
ELA 82 -0.253 0.464 
Chronic absenteeism 82 0.358 0.160 


SOURCE: NYC DOE Administrative Data (AY 2016-2017, 2017-2018, 
2018-2019). 


Table A.6 

Survey Scales from the NYC School Survey 

Scale Description 

Rigorous instruction Instruction is customized, inclusive, motivating, and aligned to the 


Common Core. High standards are set in every classroom. Students 
are actively engaged in ambitious intellectual activity and developing 
critical thinking skills. 


Supportive environment The school establishes a classroom and school culture where students feel 
safe, supported, and challenged by their teachers and peers. 


Collaborative teachers Teachers are committed to the success and improvement of their classrooms 
and schools. They have the opportunity to participate in professional 
development within a culture of respect and continuous improvement. 


Effective school leadership Principals lead by example and nurture the professional growth of teachers 
and staff, developing and delivering the instructional and social-emotional 
support that drives student achievement. 


Strong family-community ties School leadership brings resources from the community into the school 
building by welcoming, encouraging, and developing partnerships with 
families, businesses, and community-based organizations. 


Trust Everyone works toward the shared goal of improving student outcomes, 
preparing students for success in school and beyond. Across the school 
community, there is respect. School staff, parents, students, and 
administrators value each other. 


SOURCE: NYC DOE, webpage, undated-d. 


Scores for each of these scales were constructed following the Technical Guide, pub- 
lished annually by the NYC DOE (NYC Department of Education, webpage, undated-d). 
Question-level percent positive scores are first calculated for each survey question (where posi- 
tive responses are defined as those in the favorable half of response options). These question- 
level scores are averaged up to form measure-level scores, which are in turn averaged to form 
overall scale scores. Table A.7 describes values for each of the six scales. 


Three-Year Principal Retention 
We used administrative data from NYC DOE on principal service history to create a dichoto- 
mous indicator of three-year retention in the principalship. Principals were assigned a value 
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Table A.7 
Descriptive Statistics for the Third-Year School Climate Outcomes 
in Treatment and Comparison Schools 


N Mean Percent Positive SD 
Rigorous instruction 82 74.16 7.96 
Family ties 82 88.71 5.88 
Supportive environment 82 71.89 10.01 
Trust 82 86.54 7.10 
Collaboration 82 83.22 7.38 
Effective leadership 82 84.96 8.42 


SOURCE: NYC DOE administrative data (AY 2015-2016, 2016-2017, 2017-2018). 
NOTE: Summary statistics are based on percent positive responses averaged to 
form scale scores. 


of 1 if they remained in a principal position three years after initial placement, and a value of 0 
otherwise. Across all principals, the average three-year retention rate was 87.9 percent. 


Fidelity of Implementation of Program Elements 

Above we provided a detailed description of how fidelity measures were created for the five 
key components of the TISS program. We selected two measures of implementation fidelity 
to incorporate into the impact analysis to assess relationships between implementation and 
impact: 


1. Fidelity of implementation across most key components: We calculated a count of the 
number of key components of TISS that were implemented with fidelity at each school 
and assigned a 1 to schools where at least four of the five key components were imple- 
mented with fidelity, whereas individuals in schools with three or fewer key compo- 
nents implemented with fidelity were coded as a 0. 

2. Coplacement: Given the reported importance of coplacement as supporting successful 
implementation across multiple key components, we thought it would be valuable to 
separately examine its relationship with effects. 


APPENDIX B 


Additional Findings and Robustness Checks 


Confirmatory and Exploratory Impact Estimates 


Below we present our estimates for our three-year student achievement outcomes, our confir- 
matory outcomes analysis. We also provide estimates for our wide range of exploratory out- 
comes and analyses, including chronic absenteeism, school culture measures, principal reten- 
tion, one- and two-year (interim) outcomes, and analyses by cohort. 


Third-Year Estimates Detailed in the Main Report 

Table B.1 provides estimates of the effect of TISS on student achievement, absenteeism, and 
principal retention three years after placement. Table B.2 provides estimates of the effect of 
TISS on school climate responses three years after placement. As described in the report, 
results were negative and statistically significant for student achievement in ELA and four of 
our six school climate measures, indicating that placement of a TISS principal rather than a 
non-IISS principal led to lower scores on state assessments and worse climate measures. 


First- and Second-Year Outcomes 


Table B.3 provides estimates for our student and school outcomes one and two years after 
placement. We found negative statistically significant effects on student achievement in 


Table B.1 
Third-Year Impact Estimates for TISS Program Participation 
on Academic Outcomes and Chronic Absenteeism 


Outcome Estimate Standard Error 


Student achievement 


ELA -0.128 0.067 
Mathematics -0.071 0.053 
Reduced chronic —0.049* 0.024 


absenteeism 
Principal retention 0.063 0.073 


NOTE: Not shown in this table are the coefficients for variables 
representing baseline characteristics. Standard errors are 
clustered at the principal level. * signifies that estimates were 
statistically significant at the p<0.05 level. 
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Table B.2 
Third-Year Impact Estimates for TISS Program Participation 
on School Climate 


Outcome Percent Positive Standard Error 


School climate 


Collaborative teachers -0.638 2.180 
Effective leadership -1.539 2.663 
Rigorous instruction —0.662 2.304 
Family ties —0.835 1.702 
Supportive environment 2.352 2.271 
Trust -1.557 2.392 


NOTE: Not shown in this table are the coefficients for variables 
representing baseline characteristics. Standard errors are clustered at the 
principal level. None of the estimates were statistically significant at the 
p<0.05 level. 


Table B.3 
One- and Two-Year Impact Estimates for TISS Program Participation 
on All Outcomes 


After One Year After Two Years 
Outcome Est (SE) Est (SE) 
Student achievement 
ELA —0.022 -0.204 
(0.051) (0.067)** 
Mathematics —0.022 -0.092 
(0.044) (0.056) 
Reduced chronic absenteeism 0.007 -0.021 
(0.016) (0.022) 
School climate 
Collaborative teachers 2.698 -0.971 
(2.935) (2.342) 
Effective leadership -6.421 -1.570 
(3.729) (2.759) 
Rigorous instruction -1.659 -0.964 
(2.376) (1.938) 
Family ties -1.988 —0.562 
(1.467) (1.425) 
Supportive environment 2.884 —2.689 
(1.996) (2.580) 
Trust -4.114 -1.679 
(2.649) (1.973) 


NOTE: Not shown in this table are the coefficients for variables representing 
baseline characteristics. ** signifies that estimates were statistically significant 
at the p<0.01 level. 
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Table B.4 
Third-Year Associations of TISS Effects with Fidelity of Implementation 


Estimates (SE) 


TISS* Fidelity of 


TISS Implementation Implied Effect at Full Fidelity 
Student achievement 
ELA —0.053 -0.136 —0.189 
(0.061) (0.088) 
Mathematics —0.012 —0.107 -0.119 
(0.053) (0.074) 
Reduced chronic -0.019 -0.054 -0.073 
absenteeism (0.03) (0.034) 
School climate 
Collaborative teachers —0.737 0.177 —0.560 
(3.183) (3.932) 
Effective leadership -1.204 —0.601 -1.804 
(3.742) (3.655) 
Rigorous instruction 0.876 —2.756 —1.880 
(3.258) (4.083) 
Family ties —0.863 0.049 -0.814 
(2.22) (2.323) 
Supportive environment 0.001 -4.217 -4.216 
(3.402) (4.327) 
Trust -1.791 0.419 -1.372 
(3.404) (3.637) 


NOTE: Not shown in this table are the coefficients for variables representing baseline characteristics. 
Standard errors are clustered at the principal level. None of the relationships were statistically significant 
at the p<0.05 level. 


ELA after two years, indicating that students at TISS schools did not perform as well on state 
assessments as students at comparison schools. Effects on school culture measures were not 
statistically significant but were always negative, suggesting placement of TISS principals was 
leading to worse school climate responses. 


Third-Year Associations of Effects with Fidelity of Implementation 

Table B.4 provides information of the relationships between high fidelity of implementation 
and TISS effects. The estimates presented in the first column of the table represent the effect of a 
TISS principal with low implementation fidelity (fidelity on 3 or fewer program features). The 
estimates in the second column represent the difference effect between a school where TISS was 
implemented with fidelity across most components (four or five) and a school where TISS 
was implemented with fidelity across three or fewer components. Positive estimates would 
suggest that fidelity across four or five components was either less harmful or more beneficial. 
There are no statistically significant estimates in Table B.4, suggesting that there is no evidence 
that schools where TISS was implemented with fidelity across most components performed any 
better or worse than TISS schools with fidelity across fewer components. The implied effects 
at full fidelity are all negative in direction, suggesting that even with high fidelity, placement 
of TISS principals was leading to worse student and school outcomes. 
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Table B.5 
Associations Between Third-Year TISS Effects and Coplacement 


Estimates (SE) 


Implied Effect at Full 


TISS TISS*Coplacement Coplacement 
Student achievement 
ELA —0.086 -—0.089 -0.174 
(0.069) (0.098) 
Mathematics -0.013 -0.122 -0.135 
(0.064) (0.073) 
Reduced chronic -0.043 -0.011 —0.055 
absenteeism (0.032) (0.039) 
School climate 
Collaborative teachers 0.427 2.431 2.004 
(3.129) (3.106) 
Effective leadership 2.404 1.974 —0.430 
(3.795) (3.725) 
Rigorous instruction 1.946 —5.95 —4.003 
(3.419) (4.181) 
Family ties —2.221 3.161 0.940 
(2.295) (2.195) 
Supportive environment 0.582 —6.693 —6.111 
(3.155) (3.569) 
Trust -0.929 —1.432 2.361 
(3.336) (2.815) 


NOTE: Not shown in this table are the coefficients for variables representing baseline 
characteristics. Standard errors are clustered at the principal level. None of the relationships were 
statistically significant at the p<0.05 level. 


Table B.5 provides similar information of the relationship between coplacement and 
TISS effects. There are no statistically significant estimates in Table B.5, suggesting that there 
is no evidence that schools with coplaced TISS principals performed any better or worse than 
those schools without coplaced TISS principals. The implied effects at full fidelity are gener- 
ally negative in direction, suggesting that TISS schools with coplacement were underperform- 
ing comparison schools. 


Results of Comparative Interrupted Time Series 
Analyses for Baseline Trends 


Table B.6 provides estimates based on the CITS analyses. Similar to the findings from our 
primary model specification, none of these estimates are statistically significant. The estimates 
for math and chronic absenteeism are similar to the estimates from the primary specifica- 
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Table B.6 

Comparative Interrupted Time Series Design Estimates 
of Program Participation on Academic Outcomes and 
Reduced Chronic Absenteeism 


Outcome Est (SE) 


Student achievement 


ELA 0.028 
(0.160) 

Mathematics —0.053 
(0.125) 

Reduced chronic absenteeism -0.021 
(0.039) 


NOTE: Not shown in this table are the coefficients for variables 
representing baseline characteristics. None of the relationships 
were statistically significant at the p<0.05 level. 


tion in both magnitude and direction, though the standard errors are considerably larger. For 
ELA, there is a change in sign (here the estimate is positive), suggesting that accounting for 
the baseline achievement trends in ELA may be important. However, this estimate is also not 
statistically significant and is quite small in a practical sense (less than three-hundredths of a 
standard deviation). 


APPENDIX C 


Interview Protocols 


Principal Interview Protocol 


The TISS Model 
1. Can you tell us a little bit about the TISS program and what you see as the main 
features? 


Recruitment and Matching 
2. How did you first hear about the TISS program? 
3. Why did you decide to participate in the TISS program? 


4, Can you tell us about the process to match you with a leader partner? 


a. 


b. 


Cc. 


Placement 


How did the matching happen? 

Can you tell us about the interaction you had with your leader partner prior to 
being placed in a school? 

Were you satisfied with the process? 


5. How did you identify and obtain your current position? 


a. 
b. 
e 


d. 


e. 


How long did it take to find a good match/good position? 
What role did NYCLA provide? 

What support did the district provide? 

What challenges did you encounter? 

Was it easier or harder than you expected? 


6. Were you placed with your leader partner? If not, why? 


Coaching and Diagnostic 
7. Can you tell us about the coaching and how that works? 


a. 


of 


° 


How do you communicate with your coach (e.g., in person, by phone, by email, 
at meetings)? 

How often do you meet with the coach? 

Who typically initiates the meetings? 

How structured are the interactions with your coach? What do you discuss? Why? 
What do you do during the meetings? 

How often do you meet with the coach alone? With others? 

When you do meet with others, who typically joins? 
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8. Do you find your interactions with the coach to be useful? Why or why not? 
a. Do you think that the amount of interaction is sufficient? 
b. Do you think the level of structure is appropriate? 

9. Have you encountered any challenges with the coaching? 

10. Did you use the diagnostic at all this year? In what ways? 

a. How did the diagnostic affect your decisionmaking? 
b. How did the diagnostic affect your interaction with your coach? 
c. Do you feel that the school diagnostic was helpful? In what ways? 
d. Were there any aspects of the school diagnostic process that you felt were not 


helpful? 


Context 
11. Do you get leadership support from other sources? 
a. District superintendents? 
b. Other sources? 
12. How aligned is TISS coaching with the support you receive from other sources? 
a. How aligned is TISS coaching with your evaluation? 


Immediate Outcomes 
13. Do you think you have benefited from your participation in TISS? Why or why not? 
14. How is your relationship with your leadership team? 
a. How is your relationship with your TISS leader partner? 
b. How is your relationship with other leaders in the school? 
c. Do you feel adequately supported by your leadership team? 
d. Do you feel that TISS has had an effect on the support you receive from your lead- 
ership team? The level of cohesion? 
e. What challenges have you encountered with your leadership team? 
15. Has TISS played a role in how supported you feel? 
a. What aspects of TISS are most important in providing you support? 
16. How do you feel that the TISS program has affected your practice as a principal? 


Reflections and Feedback 
17. Do you have any recommendations on how TISS could improve its program? 
18. Do you think every new principal should receive TISS? Why or why not? 
19. Do you think the TISS approach could be useful in other districts? 


Coach Interview Protocol 


The TISS Model 
1. Can you tell us a little bit about the TISS program and what you see as the main fea- 
tures? 
2. How does coaching under the TISS program differ from the traditional coaching that 
first-year principals typically receive? 


Interview Protocols 63 


Selection and Training 

How did you learn about the opportunity to become a TISS coach? 
What process did you go through to become a TISS coach? 

What training have your received? 


3: 
4, 
a 


ao TP 


When did the training(s) occur? 

What did the training(s) cover? 

How did the training differ from other coaching training you have received? 
Do you have any suggestions for how the training could be improved? 


Coaching and Diagnostic 
6. Can you tell us about the day-to-day aspects of TISS coaching? 


ds 


a. 


oF 


h. 


How do you communicate with the principal and/or leadership team (e.g., in person, 
by phone, by email, at meetings)? 

How often do you meet with the principal? 

Who typically initiates the meetings? 

How structured are the interactions? What do you discuss? Why? What do you do 
during the meetings? 

How often do you meet with the principal alone? With others on the leadership 
team? 

When you do meet with others on the leadership team, who typically participates? 
What information do you record about your coaching interactions and how do you 
log it? 

(For coaches with Cohort I principals) Are there differences in the coaching in Years 1 
and 2? 


We have heard about some unusual cases such as principals placed without a leader 
partner and principals who were placed without going through the full preservice train- 
ing. Has coaching for these TISS principals differed in important ways? 


a. 


b. 


Cc. 


a: 


Did you adapt your coaching in specific ways for these principals? 

For principals without leader partners, were there efforts around teaming with any 
of the other schools’ staff? 

For principals who did not receive the full preservice training, how did they learn 
about all of the components of TISS? 

Do you feel that these principals will be affected by the program differently? 


How do you feel that the coaching and diagnostic use have gone? 


a. 


Cc. 


a. 


e. 


Do you think that the amount of interaction is sufficient? Has the additional time 
been useful? 

How do you feel that the team aspect of coaching has gone? 

(For coaches with Cohort 1 principals) How has been Year 2 coaching compared with 
Year 1 coaching? 

What has worked best about the coaching? 


Have you encountered any challenges with the coaching? 


How does the school diagnostic process work? 


a. 


b. 
ce 
d 


. Were there any aspects of the school diagnostic process that you felt were not helpful? 


Can you explain the process? 
How did the diagnostic affect your coaching? 
Do you feel that the school diagnostic was helpful? In what ways? 
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Context 
10. What type of support do you receive from NYCLA? 
a. Probe on staff, logistical, training, technology 
b. What is the quality of that support? 
11. Do you have opportunities to provide feedback to NYCLA? Are staff responsive? 
12. How is your coaching affected by other support that principals are receiving? 
a. From district superintendents? 
b. From networks? 
13. How aligned is your coaching with the support the other support principals receive? 
14. How aligned is your coaching with the principals’ evaluation? 


Reflections and Feedback 
15. Do you think that TISS has affected your ability to support principals? In what ways? 
16. Do you think that TISS has affected principal practice? In what ways? 
17. Have there been other benefits of TISS for principals and schools? 
18. Do you have any recommendations on how TISS could improve its program? 
19. Do you think the TISS approach could help improve education in urban schools? In 
what ways? 
20. Do you think every new principal should receive TISS? Why or why not? 
21. Do you think that TISS may be more helpful for some principals than for others? Why? 


NYCLA Staff Interview Protocol 


Introduction 
1. What role have you played in designing and implementing TISS? 
2. Who else at NYCLA is involved with implementing TISS, and what roles do they play? 
a. Will you change the team next year in any way? 


History/Overview 
3. Where did the idea for the TISS program come from? 
a. Was there any data to support the need for the type of support provided by the TISS 
program: 
b. Who was involved in designing the program? 
4, What are the core, non-negotiable elements of the TISS model? 
a. What makes the TISS model different from traditional coaching first-year princi- 
pals receive? 
5. How did you decide on the specific aspects of TISS? 
Core components 
b. Restricting participants to residency program graduates 
c. Teaming prior to placement 
d. Placement in high needs schools 
e. Commitment to stay in school for 3 years 


ae 


6. What were the original goals of TISS? How have those goals changed over time and why? 
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Implementation 
7. How were principals and leader partners chosen to participate in TISS? 
8. How were principals and leader partners matched? 


10. 


It, 


12. 


13, 


14. 


a. Were you satisfied with the results of the matching? 

b. What worked about the matching process? What did not work? 
c. Are you planning to modify the matching process in any way? 
d. Did principals and leader partners interact prior to placement? 


. How were principals and leader partners placed? 


a. Were you satisfied with the results of the placement? 

b. What worked about the placement process? What did not work? 

c. Are you planning to modify the placement process in any way? 

d. (In Years 3-5) How do you plan to overcome the challenges you faced with the 
prior cohorts? 

How were coaches recruited and selected for the program? 

a. What skills and background were you looking for in candidates for the TISS coach 
position? Was that different from other coaching positions with NYCLA? 

What type of training did the coaches receive? 

When did the training(s) occur? 

What did the training(s) cover? 

How did the training differ from the typical training coaches receive? 

How much are coaches expected to be consistent versus tailoring to local needs? 

Will coaches receive regular training, or just when they first join TISS? 


mopnogres 


Do you plan to make changes to the training next year? 

How does the school diagnostic aspect work? 

a. How was the school diagnostic designed? 

b. How is the school diagnostic expected to affect the coaching principals receive? 

c. How is the school diagnostic expected to affect principal practice? 

How do you feel that the coaching is going? 

a. What aspects are working well? What aspects are not working as well? 

b. What challenges have you faced? What are you doing to overcome these challenges? 
How do you know whether the program is being implemented as intended? 


Outcomes 


15. 
16. 
7, 
18. 


What do you see as the immediate effects of TISS on principals and leadership teams? 
How do you think the program works to increase the support the principal receives? 
How do you think the program works to improve principal practice? 

How do you envision that improved support and improved principal practice will lead 
to effects on student achievement? 


Context 


ied 


What role does the NYC DOE play in the implementation of TISS? 
a. How do you communicate with the district? 

b. Who do you work with in the district? 

c. Is NYC DOE supportive of the TISS program? 
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20. 


os 


Aas 


23: 


What challenges have you encountered with the district (e.g., union contracts, performance 
evaluation structures)? 

a. What happens when these conflicts occur? 

b. How are you working to alleviate them? 

How is the TISS program aligned with other types of support principals receive? 

a. From district superintendents (evaluation)? 

b. From networks? 

Who is threatened by your program or in opposition to your model (e.g., other princi- 
pals, unions, universities)? 

Are there any state or district policy barriers to implementing your model? 


Reflections and Feedback 


24. 


23: 
26. 


27. 
28. 
29. 


To what extent have the goals of the program been met to date? To what extent have 
goals been exceeded? 

Do you think that TISS could be successful in any district? Why or why not? 

Do you think that TISS could be useful for any first-year principal? What principals or 
schools do you think might benefit more from the TISS program? 

What are the primary facilitators of the success of the program? 

What are the primary impediments to the success of the program? 

If you knew then what you know now, is there anything that you would have done dif- 
ferently in designing TISS? 


NYC DOE Administrator Interview Protocol 


Background 


1. 


Can you please describe your role as a district superintendent in NYC DOE? (We 

understand this is undergoing some change.) 

a. How long have you been a district superintendent? 

b. What is the role of the district superintendent in identifying principals for place- 

ment and/or schools in need of a new principal? 

What is the role of the district superintendent in matching principals to schools? 

d. What is the role of the district superintendent in supporting newly placed princi- 
pals? 

e. What is the role of the district superintendent in providing ongoing support? 

What is the nature of your interaction with the (NYCLA)? 

a. What programs offered by NYCLA are you aware of? 


© 


b. Have you worked with participants of each of these programs? 
c. With whom at NYCLA, how often, and on what issues do you interact? 
d. Have you ever participated in any of NYCLA’s programs? 


Leadership Development Pipeline Context 


2 


As a district superintendent, what internal and external resources are available to you to 
hire, oversee, and support principals? 

a. Principal preparation programs 

b. Coaching programs to support newly placed principal programs 
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c. Ongoing mentorship and development of principals and leadership staff 

d. Other programs to develop support leadership capacity in NYC DOE 

How has the role of external partners like NYCLA changed in recent years, if at all? 

To what degree do you view coaching for new principals as important to supporting 

leadership capacity in NYC DOE? 

How satisfied are you with the options you have for providing support to new princi- 

pals? 

a. How satisfied are you with the first-year coaching that NYCLA provides to princi- 
pals? 


TISS Program 


Ve 


What is your understanding of NYCLA’s TISS program and the services it provides? 

a. In what ways is the program intended to support improved leadership capacity in 
the district? 

To what extent have you interacted with the leadership teams participating in TISS in 

2014 and 2015? 

a. Have you been directly involved in interviewing TISS teams for open positions? 

b. Have you ever placed a TISS team or TISS principal in a position in your district? 

Please describe your views on the value of the various aspects of the TISS program: 

Intentional pairing and codevelopment of leaders as a team in preservice training 

Coplacement of leadership teams into schools 

Increased and sustained coaching for principals beyond Year 1 

Inclusion of the broader leadership team in coaching 

Intentional use of evidence to set priorities and target coaching through a diagnostic 

tool 


goo FP 


Reflection/ Wrap-Up Questions 


10. 


ls 


12. 


ies 


14. 


What features of TISS are most/least valuable to supporting the capacity of district 

leadership? 

What do you see as the facilitators/barriers to successfully implementing a program like 

TISS? 

Are there changes you had recommend to improve the implementation and effective- 

ness of the TISS program? 

a. Changes to the program 

b. Changes to district policies 

If another district was planning to develop a program similar to TISS, what recommen- 

dations would you make? 

a. Regarding the design of specific elements of the program? 

b. Regarding how the district can best support and incorporate the program into its 
broader strategy for leadership development and management? 

Is there anything else that you would like to share with us or would like us to convey 

to NYCLA? This can be off the record if you prefer, in which case we would not share 

this information with NYCLA. 
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